Systems and methods for mixed reality medical training

ABSTRACT

A method for training a student in medical condition identification and intervention. The method can include the step of providing a mixed reality viewing device, providing a computing device, providing a physical object, storing a medical condition scenario in memory of the viewing device, storing the scenario in the memory of the computing device, anchoring a virtual image of at least a portion of a virtual patient on at least a portion of the physical object, selecting the scenario, commanding the processor of the computing device to initiate displaying the virtual image in a selected one of a plurality of stages, controlling one or more states and one or more properties of the physical object, assessing the student&#39;s identification of the one or more virtual critical cues; and assessing the student&#39;s identification of the one or more physical critical cues.

REFERENCE TO RELATED APPLICATION

The present application is a U.S. non-provisional application thatclaims the priority benefit of U.S. provisional patent application Ser.No. 62/483,171, filed Apr. 7, 2017, and hereby incorporates the sameapplication by reference in its entirety.

TECHNICAL FIELD

Embodiments of the technology relate, in general, to augmented and mixedreality technology, and in particular to augmented and mixed realityapplications for medical training.

BACKGROUND

Between World War II and the beginning of modern conflicts in Iraq andAfghanistan, few improvements in combat medicine were made. Practicescarried out on the battlefield during conflicts in Europe wereindistinguishable from the procedures done in the jungles of Vietnam. Infact, deaths from controllable extremity hemorrhage during the VietnamWar and the beginning of Operation Enduring Freedom were virtuallyidentical, claiming the lives of 7.9% of casualties. Many of the lessonslearned in casualty care, including the use of tourniquets, analgesicsand plasma, were not implemented into best practices and standards ofcare for battlefield providers. In 1993, under the authorization of theUnited States Special Operations Command, a study of combat casualtyprehospital practices was conducted. Three years later, the findings ofthat study led to a revolution in combat medicine through the creationof Tactical Combat Casualty Care, or TCCC. The report concluded thatcorrection of extremity hemorrhage, tension pneumothorax, and airwayobstructions was not only feasible in the combat environment, but alsosuccessful in mitigating preventable deaths.

Implementation of TCCC principles across all branches of the Departmentof Defense (DoD) has been highly effective. In 1996 Col. StanleyMcCrystal, then 75th Ranger Regimental Commanding Officer, mandated TCCCtraining for all personnel assigned to the Regiment. As a result, noneof the Rangers killed in action died as a result of conditions addressedby TCCC preventable death protocols (Holcomb, 2009). Other commands haveseen similar improvements in casualty survival rates, including the101st Airborne Division, and several Special Operations Units.Recognizing both the impact of TCCC on survival rates, and the need forincreased access to higher levels of care, Secretary of Defense RobertGates mandated the extraction of wounded personnel within one hour ofinjury, shortening the time between application of TCCC and surgicallevel care. Building on these advances, there is an opportunity tocreate enhancements to combat medic training, particularly in the areaof diagnosing and treating severe trauma.

Medical mannequins are well designed for practicing interventions formedical training, including for TCCC. Medical mannequin manufacturerscontinue to add features to provide mannequins with lifelike features.

However, more lifelike features in medical mannequins cannot aloneaddress the problem of better medical training for practicinginterventions. For example, medical mannequins cannot show shifts inskin tone with medical conditions, due to the materials used for theskin of the mannequins. Likewise, medical mannequins cannot show changesin cognition, such as eye or mouth changes with changing medicalconditions. Mannequin eyes, for example, cannot signal confusion due toa prescribed underlying medical condition.

Accordingly, there is a continuing unaddressed need for systems, methodsand apparatus for improved training for medical interventions.

Additionally, there is a continuing unaddressed need for systems,methods and apparatus for improving combat medical training, such asexemplified in TCCC.

Further, there is a continuing unaddressed need for systems, methods,and apparatus for presenting to a medical trainee a medical mannequinthat can exhibit subtle visual, audible, and/or tactile cues withrespect to changing medical conditions.

Further, there is a continuing unaddressed need for systems, methods,and apparatus for presenting to a medical trainee a medical mannequincombined with peripheral medical equipment useful for a medical traineein making medical decisions.

Finally, there is a continuing unaddressed need for systems, methods,and apparatus for medical training on mannequins that can be easilytransported, administered, monitored, and reviewed.

SUMMARY OF THE DISCLOSURE

A method for training a student in medical condition identification andintervention is disclosed. The method can include the step of providinga mixed reality viewing device, providing a computing device, providinga physical object, storing a medical condition scenario in memory of theviewing device, storing the scenario in the memory of the computingdevice, anchoring a virtual image of at least a portion of a virtualpatient on at least a portion of the physical object, selecting thescenario, commanding the processor of the computing device to initiatedisplaying the virtual image in a selected one of a plurality of stages,controlling one or more states and one or more properties of thephysical object, assessing the student's identification of the one ormore virtual critical cues; and assessing the student's identificationof the one or more physical critical cues.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be more readily understood from a detaileddescription of some example embodiments taken in conjunction with thefollowing figures:

FIG. 1 depicts a schematic illustrating a system according to oneembodiment;

FIG. 2 depicts a flow chart for creating and running a training scenarioaccording to one embodiment;

FIG. 3 depicts a screenshot of from the view of a user setting aregistration point;

FIG. 4 depicts a screenshot of from the view of a user after setting aregistration point;

FIG. 5 depicts a screenshot of from the view of a user setting aregistration point;

FIG. 6 depicts a screenshot of from the view of a user after setting asecond registration point;

FIG. 7 depicts a screenshot of from the view of a user afterregistration of a virtual patient image is complete according to oneembodiment;

FIG. 8 depicts an example of skeletal tracking according to oneembodiment;

FIG. 9 depicts an inertial measuring unit being used to track limbmovement according to one embodiment;

FIG. 10 depicts schematically one method of placing inertial measuringunits on a mannequin according to one embodiment;

FIG. 11 depicts a schematic illustrating embodiment method of trackingmannequin limb movement by the use of inertial measuring units accordingto one embodiment;

FIG. 12 depicts a screenshot from a command and control communicationdevice according to one embodiment;

FIG. 13 depicts a screenshot from a command and control communicationdevice according to one embodiment;

FIG. 14 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 15 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 16 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 17 depicts a screenshot from a command and control communicationdevice according to one embodiment;

FIG. 18 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 19 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 20 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 21 depicts a screenshot from a command and control communicationdevice according to one embodiment;

FIG. 22 depicts a screenshot from a command and control communicationdevice according to one embodiment;

FIG. 23 depicts a screenshot showing a view from a smartglass accordingto one embodiment;

FIG. 24 depicts a chart showing a list of various features, uses, andsystems that can be incorporated into embodiments; and

FIG. 25 depicts a chart showing an embodiment of an organization of asystem according to one embodiment.

DETAILED DESCRIPTION

Various non-limiting embodiments of the present disclosure will now bedescribed to provide an overall understanding of the principles of thestructure, function, and use of the medical training systems andprocesses disclosed herein. One or more examples of these non-limitingembodiments are illustrated in the accompanying drawings. Those ofordinary skill in the art will understand that systems and methodsspecifically described herein and illustrated in the accompanyingdrawings are non-limiting embodiments. The features illustrated ordescribed in connection with one non-limiting embodiment may be combinedwith the features of other non-limiting embodiments. Such modificationsand variations are intended to be included within the scope of thepresent disclosure.

Reference throughout the specification to “various embodiments,” “someembodiments,” “one embodiment,” “some example embodiments,” “one exampleembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with any embodimentis included in at least one embodiment. Thus, appearances of the phrases“in various embodiments,” “in some embodiments,” “in one embodiment,”“some example embodiments,” “one example embodiment, or “in anembodiment” in places throughout the specification are not necessarilyall referring to the same embodiment. Furthermore, the particularfeatures, structures or characteristics may be combined in any suitablemanner in one or more embodiments.

Described herein are example embodiments of computer-based systems,methods and apparatus training medical responders and for determiningaccurately how well medical professionals are trained. In general, itwill be apparent to one of ordinary skill in the art that at least someof the embodiments described herein can be implemented in many differentembodiments of software, firmware, and/or hardware. The software andfirmware code can be executed by a processor or any other similarcomputing device. The software code or specialized control hardware thatcan be used to implement embodiments is not limiting. For example,embodiments described herein can be implemented in computer softwareusing any suitable computer software language type, using, for example,conventional or object-oriented techniques. Such software can be storedon any type of suitable computer-readable medium or media, such as, forexample, a magnetic or optical storage medium. The operation andbehavior of the Embodiments of the system and method can be describedwithout specific reference to specific software code or specializedhardware components. The absence of such specific references isfeasible, because it is clearly understood that artisans of ordinaryskill would be able to design software and control hardware to implementthe embodiments based on the present description with no more thanreasonable effort and without undue experimentation.

Moreover, the processes described herein can be executed by programmableequipment, such as computers or computer systems and/or processors.Software that can cause programmable equipment to execute processes canbe stored in any storage device, such as, for example, a computer system(nonvolatile) memory, an optical disk, magnetic tape, or magnetic disk.Furthermore, at least some of the processes can be programmed when thecomputer system is manufactured or stored on various types ofcomputer-readable media.

It can also be appreciated that certain portions of the processesdescribed herein can be performed using instructions stored on acomputer-readable medium or media that direct a computer system toperform the process steps. A computer-readable medium can include, forexample, memory devices such as diskettes, compact discs (CDs), digitalversatile discs (DVDs), optical disk drives, or hard disk drives. Acomputer-readable medium can also include memory storage that isphysical, virtual, permanent, temporary, semi-permanent, and/orsemi-temporary.

A “computer,” “computer system,” “host,” “server,” or “processor” canbe, for example and without limitation, a processor, microcomputer,minicomputer, server, mainframe, laptop, personal data assistant (PDA),wireless e-mail device, cellular phone, pager, processor, fax machine,scanner, or any other programmable device configured to transmit and/orreceive data over a network. Computer systems and computer-based devicesdisclosed herein can include memory for storing certain software modulesused in obtaining, processing, and communicating information. It can beappreciated that such memory can be internal or external with respect tooperation of the disclosed embodiments. The memory can also include anymeans for storing software, including a hard disk, an optical disk,floppy disk, ROM (read only memory), RAM (random access memory), PROM(programmable ROM), EEPROM (electrically erasable PROM) and/or othercomputer-readable media. Non-transitory computer-readable media, as usedherein, comprises all computer-readable media except for a transitory,propagating signal.

In various embodiments disclosed herein, a single component can bereplaced by multiple components and multiple components can be replacedby a single component to perform a given function or functions. Exceptwhere such substitution would not be operative, such substitution iswithin the intended scope of the embodiments. The computer systems cancomprise one or more processors in communication with memory (e.g., RAMor ROM) via one or more data buses. The data buses can carry electricalsignals between the processor(s) and the memory. The processor and thememory can comprise electrical circuits that conduct electrical current.Charge states of various components of the circuits, such as solid statetransistors of the processor(s) and/or memory circuit(s), can changeduring operation of the circuits.

In one example embodiment, augmented reality can be used to evaluate auser's aptitude under real-life appearances and changing scenarios. Insome embodiments, mixed reality can be used to evaluate a user'saptitude under real-life appearances and changing scenarios. In someembodiments, marker and markerless systems are used in the system andmethod described herein.

The examples discussed herein are examples only and are provided toassist in the explanation of the apparatuses, devices, systems andmethods described herein. None of the features or components shown inthe drawings or discussed below should be taken as mandatory for anyspecific implementation of any of these the apparatuses, devices,systems or methods unless specifically designated as mandatory. For easeof reading and clarity, certain components, modules, or methods may bedescribed solely in connection with a specific figure. Any failure tospecifically describe a combination or sub-combination of componentsshould not be understood as an indication that any combination orsub-combination is not possible. Also, for any methods described,regardless of whether the method is described in conjunction with a flowdiagram, it should be understood that unless otherwise specified orrequired by context, any explicit or implicit ordering of stepsperformed in the execution of a method does not imply that those stepsmust be performed in the order presented but instead may be performed ina different order or in parallel.

Deployed combat medics need cost-effective, hands-on refresher trainingthat supports identification and treatment of common causes ofpreventable death in the field such as tension pneumothorax and airwayobstruction. This training in most cases should be guided byinstructional design principles and demonstrably improve learningoutcomes. Recent advances in Augmented Reality (AR), mixed reality (MR),mannequin technology, and macrocognition measures focused on expertperformance offer valuable opportunities to meet these needs.

The system and method disclosed herein can include a method for traininga student in medical condition identification and intervention. That is,the system and method aids in teaching not only identification of visualcues in a patient, but also allows physical intervention steps to beperformed in response to the perception of visual cues. Cues can bevisual presentations to a user of a medical condition, state, orproperty, and can be predetermined for purposes of training assessmentas “critical cues”. In the present disclosure the system and method isdescribed with reference to a “trainee,” who is alternately referred toas a “student,” “learner” or the like herein. Likewise, the system andmethod can be utilized by a trainer, who can be a teacher, instructor,assessor, or the like.

Referring to FIG. 1, in an embodiment of the present disclosure, asystem 100 for medical training can include mixed reality wearablecomputer glasses, referred to generally as a mixed reality viewingdevice, which can be an augmented reality viewing device, a virtualreality viewing device or a mixed reality viewing device. In theembodiments disclosed herein the mixed reality viewing device can behead mounted and is referred to as herein as an XR device 102. Thesystem 100 can also include a command and control computing device 104.The mixed reality XR device 102 can be communicatively connected to thecommand and control computing device 104, for example in a client serverbased relationship, by first communication link 108 in any suitablemanner, including wired and wireless. The system for medical trainingcan also optionally include a data capture computing system 110 that canbe communicatively connected to the XR device 102 by secondcommunication link 112 in any suitable manner, including wired andwireless. All apparatus components can be communicatively connected bywireless network 114 that can be a LAN network, including a 5G network.

XR devices can collect information from internal or external sensors andcan control or retrieve data from other instruments or computers. XRdevices can support wireless technologies like Bluetooth, Wi-Fi, andGPS. A XR device 102 can be a Microsoft HoloLens® head-mounted display.The HoloLens® can feature an inertial measurement unit (IMU) (whichincludes an accelerometer, gyroscope, and a magnetometer), four“environment understanding” sensors (two on each side), anenergy-efficient depth camera with a 120°×120° angle of view, a2.4-megapixel photographic video camera, a four-microphone array, and anambient light sensor. HoloLens, through the use of a HolographicProcessing Unit (HPU), can use sensual and natural interfacecommands—gaze, gesture, and voice—sometimes referred to as “GGV”,inputs. Gaze commands, such as head-tracking, allows the user to bringapplication focus to whatever the user perceives. HoloLens features IEEE802.11ac, Wi-Fi, and Bluetooth wireless connectivity. Moreover, aMicrosoft HoloLens App for Windows 10 PC's and Windows 10 Mobile devicescan allow developers to run apps, use a phone or PC keyboard to typetext, view a live stream from the HoloLens user's point of view, andremotely capture mixed reality photos and videos.

The command and control computing device 104 can be a tablet computer.In an embodiment, the table computer can be a Microsoft Surface® tablet.An instructor can use the command and control computing device 104 toremotely control the training experience, including on the smartglass inuse by a trainee.

The first communication link 108 can be a wireless link, for example viaBluetooth connectivity, and can transmit to the XR device 102 controlmessages and variables, such as strings, int image files, bool macrosand the like. First communication link 108 can also transmit from the XRdevice 102 to the command and control computing device 104 mixed realityclient (MRC) data including JavaScript Object Notation (JSON) data andMRC status variables, such as strings, int image files, bool macros andthe like.

The data capture computing system 110 can be a computer, including adesktop or laptop computer using any suitable operating system capableof running the system and method described herein. In an embodiment datacapture computing system 110 can be a laptop computer running Windows®software, including Windows 10®. The second communication link 112 cantransmit MRC data and control messages to the XR device 102 and canreceive variables, such as strings, int image files, bool macros and thelike.

Network 114 can be any of known networks, cloud computing, Internet,Intranet, or other communication networks as is known in the art.

Embodiments described herein can utilize an augmented reality (AR) witha physical object. The physical object can be a mannequin 116 which canbe in the field of vision 118 of the smartglass 102. In an embodiment,the mannequin can be a trauma mannequin used in conjunction with thesystem disclosed herein to better simulate a real world trainingenvironment. The mannequin can be a substantially complete model of ahuman being. The mannequin can also be a “part task” object, such asonly the chest and abdomen of a human, or only a limb of a human. Thephysical object can also be in the shape of an animal, and the mannequincan be an animal mannequin useful, for example, in veterinary training.

The physical object, e.g., mannequin 116, can comprise a processor and awireless transceiver, and comprise moveable internal components. Theprocessor can have memory in which is stored one or more states and oneor more properties of the physical object. The moveable internalcomponents can be responsive to programmed and real-time commands andcan be responsive to wireless communications from either the command andcontrol device 104 or the XR device 102. In an embodiment, the mannequincan be a smart mannequin. In an embodiment, the mannequin 116 can have aflexible outer skin, for example, flexible vinyl or silicone, and themoveable internal components are configured to simulate skin movement ona human, such as would be observed as breathing, vein extension, pulse,swallowing, limb movement, and the like.

In an embodiment, the mannequin 116 can be a TOMManikin® from InnovativeTactical Training Solutions, Inc. TOMManikin is a trauma mannequinsimulator designed for both classroom settings and field trainingscenarios. Multiple appendages can present a variety of wounds includinggun shot wounds, blast wounds, and burns to provide full missionprofiles in combat scenarios.

Embodiments of the system and method can provide or utilize intelligenttutoring instructional expertise, clinical expertise (e.g., emergencymedicine) and rigorous evaluation capabilities, including face validscenario design and novel macrocognition performance measures.

Although the power of AR for creating effective simulation-basedtraining is intuitively appealing, it can be important to assess theimpact of the training to determine whether the desired effects areachieved. Assessing training can be complex, particularly for learningobjectives focused on macrocognitive skills such as problem detection,recognition, assessment, and sensemaking. Embodiments described hereincan address these key technical and instructional challenges.

Example embodiments of the system and method can map AR images ontophysical mannequins. Example embodiments described herein can integrateAR with mannequins, including “smart” mannequins having the capabilityof being communicatively connected to computer devices, to provide ascenario-based, interactive learning experience. Embodiments of thesystem and method can include the development of scenarios, e.g.,medical conditions presented with visual and physical cues, that canfacilitate the acquisition and objective assessment of criticalmacrocognitive skills including, for example, sensemaking to accuratelydiagnose patients, problem detection, recognition of subtle cues, andtreatment planning in dynamic settings. Embodiments of the system andmethod can leverage strategies from instructional systems design (ISD)and naturalistic decision making (NDM) literatures to provide a rigorousevaluation study.

An augmented reality (AR)/mixed reality (MR) computer system can executesoftware for integrating ISD and macrocognitive measures as described inmore detail below. The AR computer system can run on any suitablecomputing system, such as a dedicated server, a user computer or server,multiple computers, a collection of networked computers, a cloud-basedcomputer system, a web-based computer system, or from a storage device,for example. One or multiple processing units, such as centralprocessing units and/or graphics processing units, may performinstructions stored in memory to execute the processes described herein.

An AR/MR computer system in accordance with the present disclosure canbe accessed via any suitable technique, such as a web-browser such asSAFARI, OPERA, GOOGLE CHROME, INTERNET EXPLORER, or the like executingon a client device. In some embodiments, the systems and methodsdescribed herein can be a web-based application or a stand-aloneexecutable. Additionally, in some embodiments, the systems and methodsdescribed herein can integrate with various types of learning systemsdirected to specific conditions, such as tension pneumothorax and airwayobstruction, and the like. Any suitable client device can be used toaccess, or execute, the AR/MR computing system, such as laptopcomputers, desktop computers, smart phones, tablet computers, gamingsystem, and the like.

Systems and methods described herein may generally provide a realisticaugmented reality environment for users (e.g., simulation of a varietyof medical conditions) to educate and train caregivers. Interaction withthe AR/MR computer system may include, without limitation, keyboardentry, writing from pen, stylus, finger, or the like, with a computermouse, or other forms of input (voice recognition, etc.). The augmentedview may be presented on a mannequin, partial mannequin, tablet,desktop, phone, board, paper, open space, or the like. In oneembodiment, the user may interact with an augmented view of a mannequinby operating a tablet displaying the augmented view. In this embodiment,the user may receive real-time feedback, or at least near real-timefeedback, based upon their interaction with the mannequin. The AR/MRcomputer system can be a personal computer, one or multiple computers inserver-type system. User interaction with the AR/MR computer system maytake place in any of a variety of operational environments, such as awork setting or a home setting, and with one or more users interactingwith the system at a given time.

There is a tendency to focus training on skills that are easilymeasured. Medics, however, often find themselves in situations thatcannot be reduced to a single, clear, right answer. Important ‘criticalthinking’ skills include knowing what to pay attention to, rapidlyrecognizing critical conditions, and using sensemaking strategies todevelop and test hypotheses when faced with uncertainty. Although theNDM literature offers recommendations for using scenario-based trainingto train and evaluate these macrocognitive skills, few real-worldexamples exist. Advances in AR technology provide an opportunity tobuild and test these envisioned training applications.

NDM theories describe how people really make decisions intime-pressured, complex situations, and how they use mental frames todrive information-seeking activities and make sense of complexsituations. Embodiments described herein can incorporate the RecognitionPrimed Decision (RPD) model and the data frame model of sensemaking, forexample. The RPD model can provide important insight into skilledperformance and the educational elements that facilitate skillacquisition. Embodiments described herein can leverage findings from theNDM and cognitive expertise literatures to deliver a training approachthat encourages learners to engage in self-directed learning by activelyprocessing information, developing their own interpretations,explanations, and hypotheses, and articulating their own thinking.Example components of recognition skills training can include:

First-Hand Experience.

Learners can benefit from being actively involved in experiencing (i.e.,seeing, hearing) and interpreting cues in the environment. Learning canbe further enhanced if cues are embedded in realistic scenarios. Toencourage active engagement, learners can be asked a series of questionsat key decision points in each scenario. These questions/commands canhelp the learner and can include, for example: “Notice important cues(symptoms), and avoid being distracted by dramatic, but irrelevantcues”, “Generate a preliminary hypothesis (working diagnosis) based onthe information available”, “Articulate what information is needed toconfirm or disconfirm the hypothesis (confirmed diagnosis)”, “Envisionan appropriate course of action (treatment plan)”, and “Evaluate thesituation in terms of risk for each decision point”.

Expert Model.

Research suggests that exposure to an expert model can accelerate skillacquisition. After completing each scenario, learners can be presentedwith their own responses juxtaposed with expert responses to questionsfor each decision point.

Reflection.

Expertise literature suggests that a key difference between people whobecome experts and those who do not is reflection. Experts are believedto reflect back on their experiences and speculate how the situation mayhave gone differently. They extract lessons learned that they carryforward to future experiences. By asking questions at each decisionpoint and then providing an expert model, learners may develop a habitof reflecting on each challenging situation they encounter, whether itoccurs in a training setting or in the real-world.

Feedback.

Feedback can be critical to skill acquisition, yet it is difficult for ateacher to provide real-time feedback to individual learners because oftime constraints. Innovations such as embodiments of the trainerdescribed herein can provide feedback to learners immediately followingeach problem scenario. Because feedback is based on the expert model,key aspects of training can be standardized across training settings andinstructors. In addition, the proposed training can allow for compressedtime scales, so the learner can view the outcome of a particularintervention (or a delayed or absent intervention) that would normallyoccur hours or days into the future.

Scaffolding.

Moving learners progressively toward stronger understanding and,ultimately, greater independence in the learning process can be a keytenet of an NDM-based instructional approach. To scaffold appropriately,novices can receive on-the-spot feedback and hints when they fail torecognize critical cues. More experienced learners may be allowed tofail and then can be exposed to expert choices and rationale at thecompletion of the scenario to encourage reflection.

Motivation.

Motivation can be a key component of skill acquisition. It is often notenough for learners to go through the motions, they must often bemotivated to learn and apply the learning. This can be referred to as“deliberate practice”. The proposed training may motivate learners todeliberately practice in anticipation of applying skills in real-worldemergencies. In addition, Embodiments of the system and method can usestrategies for engagement used in the computer gaming industry, (i.e.,use of a dashboard that displays own performance in comparison topeers).

AR and Smart Mannequins.

Augmented reality (AR) has been used successfully in the marketing andentertainment industries to bring products and simulated experiences tolife. AR technology is a powerful way to provide the experience offirst-hand exposure to realistic perceptual cues (e.g., visual, haptic,and auditory stimuli) in situations that might ordinarily be verydifficult to replicate. This makes AR an increasingly viable tool todeliver occupational training in complex environments. While virtualreality (VR) immerses a person in a virtual environment, AR allows theusers to see the real world with virtual objects superimposed. Therelatively low cost and extreme flexibility of AR applications makesthem practical and portable.

AR may offer important advantages over existing training technologies.Mannequin systems have long been a part of Combat Life Saver trainingand their technology continues to evolve to support increasinglysophisticated medical training. Mannequins alone may provide excellentopportunities to build psychomotor skills required to administercritical interventions. But these systems are not infinitely scalable.They are specifically designed to present a relatively narrow range ofdiagnostic cues and to allow for a modest number of therapeuticinterventions. Because they are fixed hardware tools, mannequin systemsalone may lack the ability to represent certain perceptual cues that canbe vital to appropriate diagnosis and treatment of some conditions.

For example, using physical mannequins alone, a trainee may not bepresented with perceptual cues critically important to intervention,including such cues as: healthy versus atypical breath sounds; healthyversus asymmetrical rise and fall of the chest; healthy versus jugularvein distention; healthy versus atypical skin tone; healthy versusswollen airway; properly placed versus kinked needle decompression; and,properly placed versus leaky occlusive dressing. These and otherperceptual cues can be modeled and used for training with the system,method and apparatus disclosed herein.

AR-based training systems represent an important opportunity to overcomethese challenges. The robust visualization capability of AR allows forthe representation of cues, including what can be predetermined ascritical cues, that is not available in either mannequin systems orlive-tissue models. AR also makes it possible to more easily presentcues that represent injury escalation over time. Finally, softwareupdates can be rapidly developed to incorporate new conditions, newcorresponding cues and new scenarios to train associated skills. ARsolutions may eventually be a cheaper, more scalable and more robustform of training than current training approaches.

Embodiments described herein can be used, for example, with any suitabletrauma mannequin simulator designed for classroom or field settings.Such mannequins may be capable of presenting a variety of woundsincluding gunshot wounds and blast and burn injuries to provide fullmission profiles in combat scenarios.

AR may be particularly effective as a training tool in situations wherecreating and leveraging a real-world training environment is toodifficult, too dangerous, or too expensive. To successfully deliver anAR training tool that integrates recognition and assessment tasks withinterventions, realistic AR cues can be mapped onto physical mannequinso that the learner is able to directly perceive subtle changes in thepatient condition (e.g., asymmetric rise and fall of chest cavity), takeappropriate action (e.g., needle decompression), and then reassess. Toprovide this sort of interactive learning experience, embodiments of thesystem and method can incorporate target tracking, sensory input andcontrol, and the display capabilities of advanced graphics processingunits (GPUs). The following sections describe technologies that can beused in accordance with embodiments of the system and method describedherein.

AR Target Tracking.

For AR target tracking, systems can utilize computer vision frameworks.Such frameworks can support a number of visual tracking methodologies,including 2D fiducial marker tracking, 2D image tracking, human face andskeleton tracking, 3D object tracking, and environmental motiontracking, for example. Operational challenges facing visual tracking mayinclude operating in poor lighting conditions, dealing with fast-movingtargets, organic/deformable objects, and view occlusion, where somethingblocks part of the view of an AR target for a period of time. Approachesand combinations can be used to overcome these challenges. For instance,3D object tracking combined with environmental motion tracking may beused to minimize problems of view occlusion. Using dynamic occlusionreasoning with sensed 3D geometry may solve problems with occlusion seenin current tracking approaches. It is contemplated that mobile wearableproducts powerful enough to run these programs for extended periods oftime may be advantageous. To address problems with lighting, theaddition of IR emitters/sensors can help to mitigate the limitations ofusing AR in a darker environment. As for fast-moving objects, advanceswith CPU and graphic processing unit (GPU) speeds, combined withimprovements in photosensor and other visual sensor capabilities, mayimprove tracking performance.

Sensory Input and Control.

User input on mobile AR platforms can leverage existing touchscreen andvoice control approaches. Simple to intermediate voice control schemesare being employed on consumer smartphones, tablets, desktop systems, aswell as Internet-of-Things connected appliances and platforms such asAmazon Echo. Speech commands are being used for hands-free control ofmany systems' base functionality, as well as simple speech to text andtext to speech capabilities. Embodiments utilizing machine-learning canprovide robust language analysis and better context recognition offree-form speech input. Deep learning systems are contemplated, but mayrequire more processing power than current tablets and smartphones candeliver, relying on cloud-based systems, or specialized high-performancecomputers. Embodiments of the system and method can be associated withemerging wearable AR platforms, such as the Microsoft HoloLens, wheregesture control can be provided through a hand-tracking system thatmakes use of a depth camera and an onboard computer vision system.Additionally, Embodiments of the system and method can support trackingthe user's position and orientation in 3D space, with six degrees offreedom. Haptic feedback for mobile AR are contemplated and can take theform of a wireless or wired peripheral.

GPU and Display Capabilities.

Embodiments of the system and method can incorporate camera-enabledmobile devices, such as smartphones and tablets, to provide a viableplatform for AR experiences. The GPUs found on current high-end devices,when used with an optimized 3D graphics engine, can provide a high levelof visual fidelity and detail. The high resolution and ultra-brightorganic light-emitting diode (OLED) screens used in high-end mobiledevices can allow the screen to be visible in a wide range of viewingconditions, from bright to dark. OLED is generally light, flexible andallows for transparent and see-through displays, making it a viablemedium for AR display systems and AR wearables. Embodiments of thesystem and method can include sensing systems using infrared (IR) thatare incorporated into next generation smartphones to provide 3D depthmapping in support of motion tracking and area learning. Embodiments ofthe system and method can include AR eyewear devices. Embodiments of thesystem and method can include AR display systems that can provide nearlyphotorealistic real-time 3D graphics using mobile game engines likeUnity and Unreal. On high-end mobile devices, these engines can supporthigh polygon counts, with multiple levels of detail, real-time globalillumination, physically-based shading, as well as material propertiessuch as specular, reflection, bump mapping and normal mapping.Specialized shaders can be developed to create visual effects that mimicthe look of metals, glass, vapors and liquids. Physics systems can beemployed to simulate realistic phenomena, such as fluid, smoke and fire.Utilizing a 3D game engine may have additional benefits, including theability to deliver 3D spatial audio. Sound can be used to add depth,realism, and emotion to an AR experience, where users can listen tovirtual characters speak and hear virtual props make sounds as they usethem.

Evaluation Strategies.

Numerous sources are available that can provide valuable guidanceregarding what constitutes meaningful evaluation of traininginterventions. When applying this evaluation framework, the focus caninclude survey responses regarding attitudes, test scores assessingknowledge, and directly observable performance measures. These ISD-basedcomponents can establish a critical baseline for evaluation. Embodimentsof the system and method can be used to augment traditional ISDevaluation strategies with macrocognitive metrics and measures, whichcan place a strong emphasis on expertise development. Table 1 belowadapts 6 levels of scenario description identified for evaluating theimpact of decision support technology for use in designing face validscenarios to objectively assess the impact of the training.

TABLE 1 Levels of scenario description for evaluation Levels of ScenarioDescription Indicators of Sufficient Integrity Surface validityProfessionals representing the target user population are engaged in thescenario activities and consider them face valid. Participants arewilling to perform tasked activities during the evaluation. Participantscan easily explain why scenario elements are appropriate for their worksetting. Participants recommend participation in the training tocolleagues. Representative The scenarios reflect the range ofcomplicating situations that experts are complexity likely to face onthe job. Model of The scenario reflects complex situations where thetraining is expected to support improve performance. The scenarioreflects a range of complexity to explore the boundaries of typicalityversus anomaly - where common heuristics are likely to break down. Thescenario supports a qualitative, holistic assessment by participants andevaluation team members about whether the training addresses skills thatwould be useful in their real-world setting. Scenario There is a rangeof performance across study conditions, making it difficulty possible toevaluate the contribution of the training to skill acquisition.Performance The scenarios enable objective observers to reliablydetermine the quality observability of cognitive and collaborativeperformance via externally observable indicators (e.g., participantutterances or observable actions) Value/impact The scenarios reflectcomplex situations where improvements in potential performance would beanticipated to reap significant benefits to the organization (e.g., moreaffordable, more portable training) as well as the end user (e.g., moreconfident, more prepared)

Macrocognitive metrics tend to be tailored to a specific situation orscenario. For example, Embodiments of the system and method can use arigor metric that focuses on elements such as appropriate hypothesisexploration, information search, information synthesis, and others inthe context of a specific scenario. Behavioral indicators related toeach element are identified for individual scenarios. Embodiments of thesystem and method can integrate macrocognitive measures with establishedISD measures to design a robust evaluation strategy.

Referring to FIG. 2, it will be appreciated that the design,development, and execution of a training program can include anysuitable methodology, such as Method 200. For example, Developing aScenario 202 can incorporate cognitive task analysis (CTA) interviewtechniques to elicit a case base of real-life incidents involving, forexample, tension pneumothorax and airway obstruction. Interviews can beconducted with experienced emergency medical services (EMS) andemergency department (ED) personnel, including experienced pararescuejumpers (PJ), ED physicians, and others identified that may be relevantor useful. The incidents gathered in these interviews can serve as thefoundation for both training and evaluation scenarios and can providecontext and details that can be used to meet the criteria described inTable 1.

CTA focuses on understanding the cognitive aspects of a particular job.Developed specifically to aid interviewees in articulating tacit aspectsof expertise, modern CTA methods emerged in the 1980s, and have beenrefined extensively over the last 30 years. CTA methods can includein-depth, incident-based interviews, and observations of expertperformance in naturalistic or simulated settings. CTA can be beenapplied across a range of military and commercial domains in the contextof both basic and applied research. Embodiments described herein canapply a specific CTA method called the critical decision method (CDM),for example. The CDM is perhaps the most well-established CTA method.First articulated in 1989, the method is based on Flanagan's CriticalIncident technique. Interviewees can be asked to recall an incident inwhich their skills were challenged. After obtaining a brief overview ofthe incident, the interviewer and interviewee can work together to builda rough timeline of the major events. From the timeline, the interviewercan then probe critical points in the incident to explore elements suchas goals that were considered during the incident; options that weregenerated, evaluated, and eventually chosen; cue utilization; contextualelements; and situation assessment factors specific to particulardecisions. In the final portion of the CDM interview, hypotheticalquestions can be used to explore errors inexperienced personnel mighthave made in similar situations, and to explore the implications ofspecific cues or events.

The system and method of the disclosure can utilize scenarios, which canbe medical conditions presented by the system and method for training.Scenarios can have one or more stages of the medical conditions. Foreach stage the system and method can present to a student traineevarious time-based symptoms and other perceptible cues, which can becritical cues for a trainee to observe and react to. Example scenariosand representative stages of time-based conditions with visual cues tobe presented to a student trainee for are as follows:

Example Scenario 1: Tension Pneumothorax

An explosive ordnance disposal (EOD) soldier was attempting toneutralize a small improvised explosive device (IED) The IED detonatedduring the deactivation attempt, causing a blast lung injury, temporaryloss of consciousness, blunt force trauma to the rib cage, and burns tothe patient's hands. The patient was thrown approximately two metersaway from the blast site, and landed on pavement. This is an activecombat situation, so a key concern is security. Patient is a male, 20-23years old, approximately 180 lbs.

Time 1: 30-60 seconds after point of injury (POI): At this point, thelearner is able to do a quick assessment of the patient, but needs tostay alert to changing combat conditions. The patient does not showsigns of massive hemorrhage, but there is some blood in his mouth. Thepatient has close to full respirations, but his inhalation volume isdecreased. He has a fast rate of breathing and is showing some signs ofbreathing difficulty. The patient is alert, but is in pain and hassuperficial and partial thickness burns to his hands. Assessment isinterrupted by active fire.

Time 2: 15 minutes after POI: The area is now secure, so the learner isable to continue assessing the patient. The learner is able to determinethat there are broken teeth in the patient's mouth, and it looks likethe patient bit his cheek and tongue (explaining the blood in the mouthseen at Time 1). The patient has bruises on his right rib cage, andthere is a slight paradoxical motion in the rise and fall of his chest.The patient is having more difficulty and increased effort of breathing.The patient is alert and talking in short sentences. The patient is nowable to be moved to a more secure area.

Time 3: 45-60 minutes after POI: The patient is now in a secure andprotected area, and the learner is able to re-assess. Since the lastassessment, the patient is breathing more rapidly and is now wheezing.He is becoming hypoxic and cyanotic. The paradoxical motion in his chestis more pronounced, and now there is an obvious flail chest segment. Thebruises to his torso are larger and have changed color. The learner isable to see unilateral rise and fall of his chest. The patient isconfused and his skin is pale.

Time 4: 3 hours after POI: The patient has not yet reached a definitivecare facility, but he is sheltered in a secure location. Uponreassessment, the learner notices that the patient is coughingexcessively with productive red sputum. The patient has decreased breathsounds and there is blood coming out of his mouth. The patient isshowing unilateral rise and fall of the chest, and his torso bruisinghas worsened. The patient shows jugular vein distention and trachealdeviation. His skin is pale and he is in and out of consciousness.

Scenario 2: Airway Obstruction

A rocket-propelled grenade (RPG) strikes the driver side front door ofan up-armored vehicle in a convoy. The driver (wearing a ballistichelmet, body armor, and gloves) was closest to the explosion andremained entrapped in the vehicle for approximately 90 seconds beforehis teammates are able to pull him from the burning wreckage. Hereceives partial and full thickness burns around his neck, jaw, andupper cheeks. The body armor protected his chest from any direct burnsor penetrating injury. The soldier's upper extremities were hit withshrapnel and debris. His gloves protected his hands, but there are ringburns around his wrist.

Time 1: Immediately following POI: The learner is able to get to thepatient shortly after he was removed from the burning vehicle. Aftergetting the patient to a secure location, the learner assesses thepatient. While there is blood on his uniform, it is from lacerationsfrom debris; there is no massive hemorrhage. The patient has singednostril hairs and the skin on his face and his lips is turning red, likehe has been sunburned. There are also small pieces of shrapnel in hisface. There are no major secretions in his mouth and his neck looksnormal. His voice is hoarse and he is coughing. The patient hasincreased respirations, but his chest movement is normal. Patient isconscious but in pain and seems to be rambling. He is sweating and haslight burns on his arms.

Time 2: 5-10 minutes after POI: The patient has become lethargic and isin obvious respiratory distress, so you reassess. You notice that thetissue along his airway is beginning to swell, and the patient iscoughing up thick carbonaceous sputum with some yellow-green phlegm andblood. The skin around the patient's mouth is becoming a darker red, andblisters are forming across his cheeks and forehead. The patient'snostrils are becoming smaller, and there is general swelling around hisface. There are more secretions in his mouth, and his pharynx isstarting to close up.

Time 3: 15 minutes after POI: Non-surgical attempts at managing thepatient's airway have failed, and the patient's condition is worsening.The tissue in his nose and mouth is significantly swollen; his lips aretriple in size. Larger blisters are starting to form on his face. Thecarbonaceous sputum and secretions in the mouth are very prominent. Thelearner hears stridor, wheezing, and wet lung sounds. The patient has anincreased effort and rate of breathing. The patient is fatigued and hasa muted cough.

Each of these example scenarios describe the trajectory of the patients'injuries if appropriate medical interventions are not applied. Thesystem, method, and apparatus of the present disclosure can modelvirtual patients that reflect these scenarios.

Method 200 can include Developing an Expert Model 204. After scenarioshave been drafted, subject matter experts (SMEs) can review eachscenario and provide feedback regarding refinements to increase realismand training efficacy, and to identify important variations to includein future scenarios. When two training scenarios and two evaluationscenarios have been finalized, each SME can be asked to work through thescenarios and respond to question probes at each decision point. Theresearch team can review SME responses and develop an expert model thatwill drive feedback for the learner at each decision point. When expertsdo not agree, agreement can be reached by consensus or feedback couldinclude a minority view.

Method 200 can include Developing a Prototype 206. Any suitableplatform, such as Unity® 3D from Unity Technologies, can be used as thedevelopment platform for the client AR application. Embodiments of thesystem and method can include using a platform that can support a widerange of existing and emerging devices, including AR platforms such asGoogle's Project Tango and Microsoft's HoloLens. Applications, such asUnity 3D apps, can be developed once and then deployed across a varietyof devices with relatively little additional work. Embodiments of thetraining technology can be available on a wearable platform such as theabove-mentioned HoloLens and/or Android-based tablets enabled withGoogle Tango 3D depth sensing technologies to support relativelyinexpensive and rapid iteration, participatory design, and testing.Example features or elements associated with Developing a Prototype 206can include the following:

Virtual Props.

Virtual props can be used in developed training scenarios. Virtual propscan be images of peripheral equipment used to depict real-world objectsand equipment that would be present in a simulation environment.Examples used in medical simulations include medical interventions suchas decompression needles, chest seals, endotracheal tubes, intravenous(IV) bags, as well as devices such electrocardiograms (EKG), pulse/oxmonitors, mobile field sonograms, and the like. All virtual assets canbe designed to scale across low-powered mobile devices all the way up tohigh-powered GPU-based gaming systems and simulation systems.

Virtual Humans.

Virtual humans can be incorporated into the system and method disclosedherein. Virtual patients can be built from high-resolution full bodyphotogrammetry scans of real humans or in any other suitable manner.Embodiments of the system and method can utilize technology used instate of the art movie visual effects to achieve the highest qualityresults. 3D artists can takes this scan data and create refined virtualpatient assets that can be used in embodiments of the simulation system.On higher-powered systems, virtual humans can utilize dynamic skeletaland facial animation, for more realistic and engaging performances. 3Dspatial audio can be supported in the AR training platform, which cansupport a virtual human's sounds and voice such that they are depictedaccurately. Custom shaders can be used to simulate the appearance ofhuman skin and sub-surface light scattering effects through thin tissue.

Virtual Patient Registration to Physical Mannequin.

As depicted in FIGS. 4-8, embodiments of the system can use mannequin116 with a virtual image registered thereon for viewing through thesmartglass 102. Virtual image registration can utilize markerlesstracking and a user initiated 2-point registration process to the lockthe virtual patient overlay to a physical mannequin 116.

As shown in FIG. 3 which is a view from a HoloLens looking at a physicalTOMManikin mannequin 116, an indicator cursor 120 appears in theviewport of the HoloLens and can be positioned on the mannequin. Cursor120 is viewed through the HoloLens and moves with the viewer's headmovement. The cursor can appear after a prompt from the system toregister the mannequin, and a voice command, such as “register patient.”In an embodiment an indicator can instruct the user to place the cursoron a predetermined position on the mannequin, such as the mannequin'spelvis. In such an embodiment, a user can then set the marker, such asby a voice command, for example, “set pelvis.”

As shown in FIG. 4, after the “set pelvis” command a virtual guide 122can appear in the HoloLens view on the mannequin's pelvis to indicatethat the first registration point is set. At this time, the user canview a message instructing her to place the cursor on a second locationof the mannequin, such as the mannequin's chest, and say “set chest.” Asshown in FIG. 5, as before, an indicator cursor 120 appears in theviewport of the HoloLens and can be positioned on the mannequin's chest.Again, the user can use a voice command, such as “register patient,” andthe second registration point can be set.

Once the two points are set for registration, a virtual grid 124 showingthe placement of the AR virtual patient image can appear overlaid onmannequin 116, as shown in FIG. 6. The user can continue to move thepelvis and chest points to more accurately properly place the AR virtualgrid, and can then give a voice command such as “finish registration.”Once registration is complete the user views a virtual patient image 132overlaid on the mannequin, as depicted in FIG. 7.

As discussed in more detail below, after registration, the user, eithera trainer or a trainee, can call up a medical scenario by stage. Forexample, a user can call up a trauma scenario by number and a stage oftrauma for that scenario by number. In an embodiment, for example,“Scenario 1” can be a scenario depicting a patient suffering fromtension pneumothorax. Once Scenario 1 is called up by a voice commandsuch as “scenario one,” the user can then set a stage of the condition,such as “Stage 1,” which can be the beginning stages of tensionpneumothorax, for example.

Once a scenario is called up, and if available, a stage, the system andmethod can lead a student through more stages with varying conditionsbeing presented by the virtual image mimicking the changes a body goesthrough in the various stages, as more fully disclosed below.

Embodiments of the system and method can utilize a mix ofmarker/markerless tracking approaches to align the virtual patient'sextremities to that of the physical mannequin. In an embodiment, one ormore fiducial markers can be utilized to anchor the virtual image ontothe mannequin. Conventionally, a fiducial marker is an object that isused in the field of view of an imaging system and which appears in theresulting image. In other words, conventional markers can be used asmarkers in images and not as markers on real-world objects. Because thedisclosed AR system and method relies on real-time registration andrendering of synthetic imagery onto the mannequin, the system can employfiducial markers, or image patterns with a known size and configurationto track the position and orientation (pose) of the mannequin, or todetermine the user's position with respect to mannequin. The fiducialserves two purposes; the first is determination of the position of theuser's vantage point with respect to the fiducial, and second isrelating the position of the user to a map or model of the environmentor item to be augmented.

Augmented reality fiducials can take two forms. According to onetechnique, a target is constructed to allow reliable detection,identification and localization of a set non-collinear points where thearrangement and location of the points is known a priori. This is calleda synthetic fiducial. The second approach is to use a set readilyidentifiable naturally occurring non-collinear patterns (or imagefeatures, for instance, a small image patch or a tattoo image) in animage that can be reliably detected, identified, localized, and trackedbetween successive changes in a camera pose. These are called naturalfiducials.

In an embodiment, the appearance of fiducial markers in images serves asa reference for image scaling. For instance, fiducial markers at knownlocations in an image can be used to determine the relative scale of theimage. Fiducial markers can also be used to make features of an imagemore visible. Fiducial markers can be embodied as tags that are placedin space or in the mannequin to assist in object recognition, objecttracking, and/or object modeling. Embodiments of the fiducial markersdisclosed herein, by way of example and not limitation, support activeinterrogation, are recognizable at comparatively longer distances, areless obvious to ordinary inspection, and/or are able to storesignificantly more data or enable access to significantly more data.

In an embodiment, the mannequin 116 is marked with two markers that canbe any image having suitable computer-identifiable characteristics, asis known in the art. In an embodiment, the markers can be images oftattoos, for example. By locking in to first one fiducial marker andthen the other and getting the registration set off of these marks,offsets and other criteria associated with registration can be set. Ifthe mannequin is moved, the system can quickly relock in on those sametwo markers, resulting in the system keeping the same configuration ofvirtual image overlaid on mannequin 116.

Using fiducial markers improves the speed of the registration processdescribed above that relies on anchoring to a spatial map. Usingfiducial markers also solves the problem of movement of the mannequin.When anchoring on a spatial map, if the spatial map changes, i.e. if youmove the mannequin, the anchor points are regenerated, essentially, andthey're no longer relevant to where the new mannequin's position is.

It is possible to combine multiple approaches to improve theregistration and tracking of a virtual object mapped to a physicalobject. For example, by leveraging 2D fiducial markers with simultaneouslocation and modeling (SLAM) techniques, a fixed location in space canbe determined. The location provided by the fiducial markers can then bedigitally anchored to an existing spatial map created using othertechniques. This is particularly useful in situations where a physicalobject may be moved from one location to another. By leveraging thepower of fiducial markers, a virtual object can quickly be relocated toanother position in 3D space simply by reacquiring the fiducialmarkers—shortening the time needed to construct an entirely new spatialmap and re-insert the virtual object. In dynamic training environments,such as tactical combat casualty care and prolonged field care, thevalue of rapid re-registration of a virtual object to a physical objectis dramatically increased.

Fiducial markers can be attached, e.g., by adhesive, to the mannequin,such that they are always in a very consistent place. The user no longerhas to look at just a blank mannequin and guess where these offsetswould be using computer vision software.

In an embodiment Vuforia® Augmented Reality Software Development Kit(SDK) can be utilized for fiducial marker systems. Vuforia uses computervision technology to recognize and track planar images (Image Targets)and 3D objects in real-time. The Vuforia image registration capabilityenables users of the system disclosed herein to position and orientvirtual objects, such as the patient virtual image, in relation to realworld objects, such as the mannequin, when these are viewed through aviewer such as the HoloLens, or the camera of a mobile device. Thevirtual object then tracks the position and orientation of the image inreal-time so that the viewer's perspective on the object correspondswith their perspective on the Image Target, so that it appears that thevirtual object is a part of the real world scene. which supportscomputer vision as well augmented reality content. Using computervision, we can use these visual reference markers that are adhered tothe mannequins to always have a consistent placement of registrationpoints. This removes the variability of these points being a little bitdifferent on the mannequin every time that a user manually registers thevirtual patient to it.

In an embodiment, one or more of the fiducial markers can be in the formof a tattoo image on part of mannequin 116. In an embodiment thefiducial markers are produced with the mannequin part such that thefiducial marker is accurately and precisely placed for repeatableimaging if mannequin parts are changed over time. In an embodiment,existing mannequins can be retrofitted with fiducial markers.

In an embodiment, the fiducial markers can be visual elements adhered tothe outside of the mannequin. In this manner, fiducial markers lower abarrier of usage and improves usability because the user can simply lookat something that they can see in their normal human field of vision,and the computer vision system can see the same thing. It lowers thebarrier and time it takes to set registration points and even tounderstand that there is something that needs to be set. Someone with alow level of knowledge can use the system, with, for example, aninstruction for them to look at an eagle tattoo on a mannequin andAmerican flag tattoo on the mannequin. A user can quickly identify andutilize the fiducial marker images to gain image registration.

In an embodiment, the system and method disclosed herein combinesspatial mapping with the fiducial marker tracking. The fiducial markerscan be used to set registration points. Once the registration points areset, they are transferred into a spatial mapping system. At this pointthe fiducial makers no longer need to be seen in order to have a virtualpatient align to the mannequin through spatial mapping.

In an embodiment, fiducial markers are not used to real time track avirtual patient to the mannequin; rather they are used forinitialization and setup of the virtual patient.

In a scenario where fiducial markers are not used on the mannequin andthe mannequin is moved, the spatial map needs to be reset. The patientimage can still be anchored at the anchor points of the fiducial markersat the old position in the room. In order to re-register or to updatethe orientation of the virtual patient to the mannequin, the userinitialize marker-tracking mode for a moment on the device, look at thefiducial markers that are incorporated on the mannequin and essentially,re-register or relock these coordinates of the updated positions of thisfiducial marker relative to this spatial map in a new area of the room.Once the system detects that we have a new pelvis point and a new chestpoint, for example, in a different area of the room where they weren'tbefore, the spatial mapping anchor points can be updated such that thevirtual patient can update its position relative to where those newcoordinate anchor points are in the spatial map, and now show thevirtual patient correctly with the updated mannequin's position usingfiducial markers.

Embodiments of the system and method can incorporate intensive computervision approaches including face and skeleton tracking to allow forreal-time dynamic registration of the virtual patient with little to nodirect involvement from the user. For example, as shown in FIG. 8,mapping a virtual patient onto a physical mannequin with moveable limbscan be enhanced by utilizing real-time skeletal tracking using 3D depthsensors. FIG. 8 is a still shot of an image in which manual movement ofthe mannequin 116 limbs can be tracked virtually using MicrosoftKinect®. Skeleton and hand tracking are currently being used in supportof Natural User Interfaces (NUIs). Microsoft Kinect® and Orbbec® areexamples commercial products that enable skeleton tracking. Skeletontracking can be used to enhance the humanoid attributes of the mannequin116 for dynamic patient registration.

As shown in FIG. 9, inertial measurement units (IMUs) 126 can beutilized to track a mannequin's limb movements. IMU's are electronicdevices that measure and report an object's force, angular rate, and insome cases, surrounding magnetic field. IMUs use a combination ofgyroscopes, accelerometers, and sometimes magnetometers. IMUs can beplaced directly on the object to be tracked. IMUs can be chained andattached to a mannequin to track the mannequin's movements and representthe movement in the virtual environment. As depicted in FIG. 9, forexample, an IMU 126 can be attached to the hand 128 of a mannequin, witha corresponding image on a screen 130 to indicate relative movement ofthe hand 128. The image of FIG. 9 is a still shot of the hand 128 beingmanipulated and the relative movement with the arm being tracked andrepresented as spots that move relative to one another. As shown, theleft spot (as depicted in FIG. 9) can be relatively stationary while theright spot on the screen (as depicted in FIG. 9) is linked and moveswith movement of the hand.

In general, as shown in FIG. 10, any number of IMU's 128 can be placedon a mannequin 116 and correlated to the system such that movement oflimbs of the physical mannequin 116 can result in the virtual imagefollowing in real time. In FIG. 10, 21 IMU's 128 (numbered) are shownplaced on a mannequin 116.

In practice, the system utilizing IMU's can be illustrated byconsidering three sensors on a mannequin, the three IMU's being locatedat positions corresponding to a shoulder, elbow, and wrist, each at afixed length of 30 cm apart. A schematic depiction of this arrangementis shown in FIG. 11. To register a sensor 128 for mixed reality, aHoloLens user can look at sensor 0 and fix a local point in 3D space.Combined with the local coordinate from the sensors 128, for example assent to the HoloLens over the network 114, the sensors could then bemapped in a mixed reality view where the virtual object can mirror themovements of the physical object.

Hands Free Operation.

The XR device 102, such as a HoloLens, of the system can supporthands-free interactions through the use of speech recognitiontechnology. The user can call out simple two- to three-word commandphrases, for example, to control every aspect of the simulationexperience. Vision and speech recognition technology can be embedded inmiddleware associated with the system and method disclosed herein.

Personalized, Automated Training.

Embodiments of the system can incorporate scaffolding techniques suchas, for example, providing a virtual grid to aid the learner invisualizing the physical asymmetric rise and fall of the chest of amannequin, thereby better visualizing a characteristic of tensionpneumothorax, for example.

Embodiments of the system can provide personalized feedback tailored tothe user. Any suitable data can be collected, such as user gaze duringspecific points in the scenario, to provide tailored feedback. Thesystem can leverage extensive intelligent tutoring experience. Anysuitable tutoring products including Readlnsight, a readingcomprehension skills tutoring system that assesses and diagnoses eachlearner's specific reading skills deficiencies and tailors instructionaccordingly, can be used. SimBionic, an award-winning visual authoringtool and runtime engine for creating complex behaviors in trainingsimulations and games quickly and easily can also be utilized.

Referring back to FIG. 3, the method 100 can include Running a TrainingProgram 108. In an embodiment, a training program can utilize the systemand method of the invention as illustrated with reference to FIGS.12-22.

As depicted in FIG. 12, after the virtual patient image 132 isregistered onto mannequin 116 as described above with reference to FIG.8, a condition can be specified from the command and control computingdevice 104, which can present a screen 140 to an instructor, a portionof which depicted in FIG. 12. As depicted in FIG. 12, screen 140 candepict system criteria such as showing that a HoloLens is connected 142,a virtual patient is registered 144, the name (or other identification)of a student 146, and a report button 148. Additionally, an instructorview of the command and control computing device 104 can present aselection of possible scenarios 150 to present via the virtual image 132and mannequin 116 to the student.

In an embodiment, a scenario such as Tension Pneumothorax and ExtremityBleeding can be selected by clicking, touching, or voice command,whereupon the virtual patient image 132 will present an appearance of aperson in a stage of that condition. For example, as shown in FIG. 13,after selecting the scenario, which can be, as discussed above, calledby number, e.g., “Scenario 3,” the instructor screen 140 can depict animage 132 of the patient on mannequin 116 (which is shown offset forclarity herein), and the instructor can choose a “stage” 152 of thecondition to be presented to the student, for example “Stage 1” as shownin FIG. 13. On another portion 154 of the instructor screen 140 can bepresented various medical conditions that a student can be expected todetect on the virtual patient in the stage shown.

From the student perspective, viewing through a smartglass, e.g., aHoloLens, a student view 160 can show a virtual patient 132 in a firststage of tension pneumothorax. As the student move about and inspectsthe virtual patient in Stage 1, for example, the student can see a chestwound 162 on virtual patient 132, as shown if FIG. 15. The student canalso see a virtual image of a peripheral device, such as a bloodoximeter 164 that can indicates conditions typical of Stage 1 of tensionpneumothorax, such as a blood oxygen level of 98 and a pulse of 124.

In like manner, the instructor can move to other stages 152, such asStage 3, which is shown in FIG. 17 on the instructor view 140, and whichshows new medical conditions 154 presented on virtual image 132.

At Stage 3, in the illustrated embodiment, for example, the student viewcan now depict a worsening chest wound 162 as depicted in FIG. 18, achanging pulse oximeter 164, as shown in FIG. 19, that provides forrealistic typical blood oxygen levels of 85 and pulse rate of 130.

Because the system, method and apparatus disclosed herein provides for aholistic practice of diagnostic skill development. The training program(e.g., at 203 of FIG. 2) method and apparatus can integrate a virtualpatient with a physical patient that permits not only conditionidentification (e.g., via changes indicated in the virtual patient) asdescribed above, but also direct physical intervention (e.g., onmannequin 116). For example, as shown at FIG. 20, a student detectingand diagnosing tension pneumothorax can perform needle decompressionphysically inserting an actual needle 170 a into the chest of thephysical mannequin 116, such as at the second intercostal space in themidclavicular line.

As shown in FIG. 21, at every stage, an instructor can see on theinstructor view 140 what the student is viewing through smartglass 102,and can indicate, e.g., at 166, successful detection of medicalconditions and other visual cues expected to be detected by a trainee.

Referring again to FIG. 2, method 200 can include a Reporting 210.Knowledge gained in Running a Training Program 208 can be incorporatedinto subsequent development plans specifying anticipated trainingeffects, system performance goals, validation methods, key scientificand technical milestones, and risk reduction activities. Such Reportingcan be used to improve the Method 200 for subsequent iterations. In anembodiment, for example, once a training session is complete, the systemand method can present a report 168, e.g., on the instructor view 140 ofthe command and control computing device 104, as shown in FIG. 22.

An evaluation study plan can include details about participants,measures, and analysis strategies, which can be linked to specificlearning objectives. In running a training program any suitableeducators or caregivers can be recruited as participants such as, forexample, emergency medical technicians (EMTs) or EMT instructors.Participants can complete the proposed training program and can completea short knowledge test, followed by two training scenarios, for example.Scenarios can be used in which the initial situation includes weaksignals that are easy to miss. For example, a tension pneumothoraxscenario may begin with a police officer who was sliced by a knifebetween the fifth and sixth rib near the mid-axillary line when breakingup a fight. The wound does not look serious and the police officerbrushes it off, saying it is “just a nick.” In this situation, it wouldbe easy to miss the possibility of developing a tension pneumothorax. Asthe scenario unfolds, the presence of a tension pneumothorax can becomeincreasingly visible. After completing the training scenarios, theparticipants can complete another knowledge test to assess changes inknowledge after the training intervention. They can complete two testscenarios in order to collect objective performance measures andcomparisons to the expert model (i.e., macrocognitive measures). Lastly,the participants can complete a usability and usefulness survey. Theusefulness survey may aid in assessing whether EMTs judge that thevirtual patient reacts as a human would in the real world, and willinclude comparison to more traditional training platforms.

It will be appreciated from the above description that the system,method and apparatus of the present disclosure can utilizeperception-action loops that develop a holistic practice of diagnosticskill development. That is, the system, method and apparatus integratesa virtual patient with a physical patient that permits conditionidentification (e.g., via changes indicated in the virtual patient) withdirect physical intervention (e.g., on mannequin 116). Diagnostic skillscan be practiced on the virtual patient and intervention skills can bepracticed on the physical patient. The process can be monitored andcontrolled by an instructor who can control the condition of the patientand assess how the trainee is responding to changing conditions. Theperception-action loop can be repeated as necessary for training. Inthis manner, trainees are able to interact with the physical environmentand virtual content seamlessly. This functionality also affords theability to inject hints and guidance to trainees as they work throughtraining scenarios.

In an embodiment, utilizing the systems, methods and apparatus describedherein, a training session can be executed according to the followingnon-limiting description where unless specified, a user can be one oreither of a trainee student or a trainer instructor.

A student user can launch an application on the XR device, which canhold one or more medical scenarios, including time-based conditions, inits resident memory and execute instructions to launch variousapplications. The XR device allows the user to see and interact withvirtual medical patients, either overlaid on medical training mannequinsor in stand-alone mode without mannequins. An XR device app utilizesnaturalistic user interfaces and interaction controls, for examplenaturalistic language commands and human gesture controls, to the allowthe user to interact realistically with the virtual patient, mannequin,and simulation systems. The simulation systems running on the XR devicecan be controlled by the student, or remotely by an instructor usingcommand and control device.

The student can initiate start of a session with a voice command, suchas “ready for training,” and wait for a network connection with thecommand and control device. The command and control device can make anetwork connection with the XR device and send and receive commandsthrough a wireless connection. The command and control device can havein its resident memory virtual patient content and can receive one ormore scenario files, e.g., from the XR device, and can instruct how toutilize graphical assets for the user. The command and control devicecan contain complete copies of the one or more medical scenarios,including time-based conditions, loaded and used on the XR device, aswell as additional creative assets used to describe the cues,interventions and cognitive scaffolding techniques of the virtualpatient simulation system to the users, including a student andinstructor.

The command and control device can launch a application from theprocessor of the mannequin, and can then launch a training applicationfrom inside the mannequin application. After registration of the virtualpatient on the mannequin the XR device can send and event message to thecommand and control device that the image is connected and anchored. Atthis point, a user can call up a list of scenarios on the command andcontrol device. Selection of a scenario sends a message to the XR deviceload patient information and sends a message back to the command andcontrol device, whereupon stage information can populate on the commandand control device. A user can then select a stage, at which time andevent message is sent to set the statefullness in the XR device to theselected stage. At this time, a trainee can be viewing visual cues onthe virtual patient, the visual cues being virtual images or physicalconditions on the mannequin. Using gaze tracking features, the trainercan visualize on the display of the command and control device what thetrainee is viewing in real time. A trainer can, for example, confirm onthe command and control device that a trainer looked at a visual cue,and “check off” the successful visualization on the command and controldevice. This process can continue through one or more stages of eachscenario, resulting in a record of the visual cues a trainee noticedand/or reacted to. If an physical intervention is being taught, thecommand and control device can be utilized to prompt, instruct, and/orrecord any intervention events.

In an embodiment, utilizing the systems, methods and apparatus describedherein, a training session can be executed according to the followingnon-limiting description, where unless specified, a user can be one oreither of a trainee student or a trainer instructor.

A user can launch an application on the XR device, which can hold one ormore medical conditions in its resident memory and execute instructionsto launch various applications. The XR device can allow the user to seeand interact with virtual medical patients, either overlaid on medicaltraining mannequins or in stand-alone mode without mannequins. An XRdevice app can utilize naturalistic user interfaces and interactioncontrols, for example naturalistic language commands and human gesturecontrols, to the allow the user interact realistically with a virtualpatient, physical mannequin, and simulation systems. The simulationsystems running on the XR device can be controlled by the student, orremotely by an instructor using the command and control computingdevice.

A user can initiate a session with a voice command, such as “ready fortraining,” and wait for a network connection with the command andcontrol device. The command and control device can make a networkconnection with the XR device and send and receive commands through awireless connection. The command and control device can contain completecopies of the same virtual patient images loaded and used on the XRdevice, as well as additional creative assets used to describe the cues,interventions and cognitive scaffolding techniques of the virtualpatient simulation system. The command and control device can receivescenario simulation state data and simulated medical patient informationfrom the XR device in real-time, as well as provide visualization toolsto help an instructor perceive and understand what the student is seeingand doing on the XR device, granting the user total control of thevirtual patient simulation experience.

The command and control software can be portable, and can be used withstand-alone software, or embedded into other software programs. Thesoftware can use middleware to bridge between existing mechanicalmannequin control systems and the simulation system running on the XRdevice. This middleware allows a virtual patient and mannequin controlsystems to synchronize during a simulation exercise, giving the userbetter holistic understanding and control of what is happening insidethe mannequin and on the XR device's virtual patient. For example,changes in the virtual patient's respiration animation can affect thephysical mannequin's motorized respiration settings. Additionally,changes in the physical mannequin's motorized bleeding can drive thevirtual patient's blood flow on the XR device. Additionally, the commandand control device can manage broader aspects of the XR device during asimulation exercise, including knowing the status of the XR device'shardware, network and software systems.

The command and control device can also allow the instructor to thetrack the student's gaze while using the XR device. This feature allowsthe instructor to know what, where and when a student is looking duringa simulation exercise. Virtual elements associated with critical cuesrelated to medical conditions can be tagged for gaze tracking. Forinstance, areas of interest on the virtual patient can be tagged tonotify the system when an area is being seen by the student. Acollection of gaze records can be generated during each training sessionand stored in the memory of a computing device operating in the system.Time-stamped gaze records can also be created, stored, and managed foreach student.

To set up a virtual patient on the XR device (i.e. “Register thePatient”) a user can look at image markers, e.g., fiducial markers,attached to the physical mannequin in predetermined locations to triggerthe computer vision system. An image detection system running on the XRdevice can detect the image markers in the real-world environment andtranslate the image markers into virtual coordinates in a simulationengine. The detected images markers can be captured and associated ascoordinates of spatial mapping points generated by the XR device toreliably lock the virtual patient relative to the mannequin. Theregistration process can be controlled by the user on the XR device withvoice commands, or by the user of the command & control system,remotely.

When the XR device user registers the virtual patient, the command andcontrol system can be notified. At this point, a user can call up a listof scenarios on the command and control device. Selection of a scenariosends a message to the XR device to load patient information andcontent. The XR device receives the message, loads the virtual patientassets, presents the first view to the XR device wearer, then sends amessage back to the command and control device, whereupon stageinformation can populate on the command and control device. A user canthen select a stage, at which time and event message is sent to set thestatefullness of the simulation running on the XR device to the selectedstage.

At this time, a trainee can be viewing visual cues on the virtualpatient, the visual cues being medical conditions or subject matterdepicted with 3D models and graphical overlays, in-context medicalanimations and visualizations, 3D spatial sound effects and voice overacting, or medical interventions practiced on the mannequin.

Using gaze tracking features, a trainer can see visualizations on thedisplay of the command and control device of what the trainee is viewingin real time. A trainer can, for example, confirm on the command andcontrol device that a trainer looked at a visual cue, and “check off”the successful visualization on the command and control device. Thisprocess can continue through one or more stages of each scenario,resulting in a record of the visual cues a trainee noticed and/orreacted to. If a physical intervention is being taught, the command andcontrol device can be utilized to prompt, instruct, and/or record anyintervention events.

In general, in embodiments of the system and method herein, it ispossible to combine multiple approaches to improve the registration andtracking of a virtual object mapped to a physical object. For example,by leveraging 2D fiducial markers with simultaneous location andmodeling (SLAM) techniques, a fixed location in space can be determined.The location provided by the fiducial markers can then be digitallyanchored to an existing spatial map created using other techniques. Thisfeature can be particularly useful in situations where a physical objectmay be moved from one location to another. By leveraging the power offiducial markers, a virtual object can quickly be relocated to anotherposition in 3D space simply by reacquiring the fiducialmarkers—shortening the time needed to construct an entirely new spatialmap and re-insert the virtual object. In dynamic training environments,such as tactical combat casualty care and prolonged field care, thevalue of rapid re-registration of a virtual object to a physical objectis dramatically increased.

In general, in embodiments of the system and method described herein,data about a patient's condition can be exchanged between the commandand control device (C&C) application and the HoloLens. This data can bemade available to be consumed by an external object, such as a medicaltraining mannequin. That data can then be used to create changes in thephysical mannequin (e.g., change respiration, heart rate, pulse oximeterlevels, etc.). Similarly, the flow of data can also work in the oppositedirection. The C&C application can also be configured to accept datafrom an external object. For example, if a physical mannequin providesstate information or property information about the patient's condition,that data could be consumed by the C&C application and transferred tothe HoloLens application to mirror the physical patient's state. Ingeneral, as disclosed herein according to context, “states” refers tointernal models used by the system to hold data, inputs and events,e.g., variables, which collectively can define the behavior of objectsin the system. As used herein according to context, “properties” are thevalues that get transferred between systems and objects. Properties can,in some examples, be thought of as variables.

The system and method described above can benefit from five adaptivetraining components, as described below.

First, a trainee can receive attention-directing hints. In anembodiment, trainees can see or hear virtual hints and guidance,immediately practice physical interventions, and receive instructorfeedback about their actions. In the one example, the hints can be heardas a voice avatar representing a more experienced medic (or physician)encouraging the learner to notice or look for something specific, i.e.,Is he bleeding anywhere else? Have you looked inside his mouth to see ifthe airway is swelling? This approach can enable the trainee to obtainfirst-hand experiences in the context of challenging cases. This featurecan be beneficial for novice and intermediate trainees as this approachprovides important scaffolding, and also retains important realism byusing a voice avatar of mentor/colleague.

Second, trainees can receive real-time auditory correction, intended toaid the trainee in conducting a thorough assessment using a relevantalgorithm. This training component incorporates real-time feedback intothe training scenario. For example, many medical schools use the Airway,Breathing, Circulation, Disability, Expose the patient, Foley and rectal(ABCDEF) algorithm, whereas combat medics may use the Massivehemorrhage, Airway, Respirations, Cardiac, Head injury (MARCH)algorithm. When a trainee deviates from an appropriate algorithm (e.g.,delays assessing airway because of distracting head injury), an auditorycorrection can remind the trainee to walk through the assessment in thecorrect order. This intervention can be beneficial for novice andintermediate trainees.

Third, a training component can include the use of 3D animation to showunderlying dynamic anatomy. For example, although a trainee may haveread descriptions of what happens inside the chest cavity when a tensionpneumothorax develops, seeing an animation of asymmetric chest breathingon a virtual patient image overlaid on a physical mannequin can haveadded training benefit. By situating instructional animations of injuryprocesses in realistic scenarios, the present system and method canenable trainees to build robust mental models and improve their abilityto visualize the implications of the injury, anticipate future states,and identify appropriate treatment options. In another example, atrainee can see a visual pointer. A visual pointer, or cue, might pointto an appropriate intervention. As compared to a voice avatar directingthe learner to look for signs of airway swelling or describingappropriate needle placement, a pointer on the anatomy can reduceambiguity for the learner, making it appropriate for beginning trainees.For example, as shown in FIG. 23, a visual cue 250 can appear in thetrainee's field of vision via a HoloLens, together with a visualindication 252 of where and how to make an intervention. Thisintervention can be beneficial for novice and intermediate trainees.

Fourth, the system can manipulate time using AR. For example, a traineecan see how a patient condition degrades over time without medicalintervention by compressing time scales. The trainee may see the virtualpatient immediately following an injury, and then at whatever timeintervals are required to demonstrate the progression of an injury. Fora tension pneumothorax, this feature might jump forward several hours,while an airway injury might unfold in a matter of minutes. The traineecan also jump back and forth in time to compare cue representation. Forexample, the learner might first notice jugular vein distention (JVD) atStage 4 of a scenario, but wonder if s/he could have noticed it earlier.The learner could then flip back and forth between Stage 2, Stage 3, andStage 4 to see how the JVD initially appeared, and how far itprogressed. This component supports feedback and first-hand experiences.

Fifth, a performance summary can be provided. The performance summarycan have three parts: an expert model, a replay, and a performancescore. For intermediate and advanced trainees, the performance summarycan highlight where trainee performance deviated from the expert modelbased on gaze tracking data and instructor input. It can then bepossible for the instructor to revisit portions of the scenario from theperspective of the trainee. This can allow the trainee to review cuesthat were missed or misinterpreted. Finally, learners can receive aperformance score in which points are awarded for appropriate actions,and subtracted for hints used and errors. The integration of thesefeatures into a performance summary can support summary feedback,reflection, and deliberate practice.

It will be appreciated that Augmented/Mixed Reality medical trainingtools can be used for any suitable applications. Embodiments of thesystem and method can include an AR-based system that delivers trainingvia Google Tango enabled tablets and Microsoft's HoloLens wearable, forexample. Embodiments of the system and method can be a Rapid-RecognitionSkills (RRS) trainer. Embodiments of the system and method can be builtto help the user learn to rapidly recognize cues that are rare and/ordifficult to replicate in the real-world. Using computer-generatedvisual and audio cues the system can be designed to strengthen thecognitive associations between cues present in rare events and thebehaviors needed to respond appropriately to them in complex anddangerous real-world situations. Because embodiments of the system canbe portable, the platform can be integrated with live training (force onforce) technology.

Embodiments of the system can use 3D depth sensing and spatial mappingtechnologies to scan the user's environment in real-time, providing theability to lock virtual content, such as patients and props, into theuser's real-world view of the physical space. Embodiments of the systemand method can allow for a highly realistic simulation of real-worldphenomena.

Embodiments of the system and method can be targeted towards medicalfirst responders. For example, a training module can incorporate apatient with tension pneumothorax, or any other suitable medicalcondition. The system can be flexibly designed to rapidly incorporateother conditions or situations of interest. Embodiments of the systemcan include any suitable conditions such as arterial bleeding, tensionpneumothorax, and airway issues.

Embodiments of the system can effectively integrate haptic and olfactorycues into the AR trainer to further increase feedback fidelity forlearners. The system can also be associated with an authoring tool tosupport scenario selection and configuration and to address softwaresecurity. FIG. 24 shows a list of various features, uses, and systems178 that can be incorporated into embodiments described herein.Applications for this training tool for use in complex health caresettings (e.g., emergency response, hospital emergency rooms andsurgical environments) are contemplated. Embodiments of the system caninclude full integration with a mannequin at the manufacturer level.Such an integrated system may create a seamless lock-on between ourvirtual patient and the physical mannequin, develop conditions andinjury states that are aligned with the scenarios already developed forthe physical mannequin and, and/or wirelessly synchronize the patientcondition with the physical mannequin so that interventions and changesin the physical mannequin are mirrored in the virtual patient.

It will be appreciated that any suitable Mixed Reality (MR) systems canincorporate the training systems described herein. Such systems canelevate existing medical simulation training through the use of mixedreality technologies that can represent high fidelity visual andauditory cues. Embodiments of the system can include a focused andintegrated learning solution for both students and instructors,providing a mixed reality virtual patient overlay that can enhancephysical medical training via mannequins. Embodiments of the system andmethod can include a collection of patient medical conditions and stagesthat the instructor can choose and control on a selectable basis in realtime or near-real time, as well as a means to capture and store thestudent's data throughout a training session.

Example Training Session

The system, methods and apparatus disclosed herein can be used in anysuitable environment, but for illustration purposes a non-limitingexample use case is provided. Certain differences in the methoddescribed in this use case are considered alternative uses and methodsinherent in the flexibility of the system, and are not to be viewed asinconsistent with the description above. Olivia, a seasoned combatmedicine instructor, is training the Wayne County SRT SWAT Team in thebasics of Tactical Combat Casualty Care (TCCC). Today the students arelearning about tension pneumothorax. A physical training mannequin isplaced on floor of the classroom, to facilitate hands-on portions of thetraining session. The students stand off to the side as the instructorprepares the simulation space.

First, Olivia makes sure the dedicated Wi-Fi LAN network is up andrunning. She launches the Command & Control (C&C) app on her tablet,starting up the network server inside the app. She carries the tabletwith her as she sets up the other devices. Next she sets up the DataCapture System (DCS), by launching it on her laptop, and clicking the“Connect C&C” button. Moments later the DCS system plays a sound lettingher know that it has successfully connected to the C&C app. She glancesat the C&C tablet and sees an indicator confirming that the DCS isconnected. She sets up the Mixed Reality Client (MRC). She puts on theHoloLens and boots it up, launching the MRC app from the main menu.After the client app launches, it plays a sounds letting her know thatit has automatically connected to the C&C app over the network. Sheglances at the C&C tablet and sees an indicator showing that both theDCS and MRC are connected to the app.

Now that all the hardware is up and running, and the devices areconnected together on the network, Olivia can register the HoloLensvirtual patient to the physical mannequin. An indicator can appear inthe HoloLens viewport prompting her to register the patient to themannequin, by saying “Register Patient.” She says “Register Patient” anda cursor appears in the viewport. The indicator can tell her to placethe cursor on the mannequin's pelvis and say “Set Pelvis.” A virtualguide can appear on the mannequin's pelvis in the HoloLens view aftershe says “Set Pelvis,” the indicator's text changes, telling her toplace the cursor on the mannequin's forehead and say “Set Head.” Oliviacan move the cursor the appropriate location and says “Set Head,” andanother virtual guide can appear on the mannequin's forehead.Additionally, a semi-transparent virtual patient guide can now appearovertop the mannequin in the HoloLens viewport. The indicator's textchanges, letting her know that she can still move the head and pelvisguides on the mannequin by positioning the cursor and saying “Set Head”and “Set Pelvis” until she's happy with the virtual patient's placement.The indicator also tells her that if she's happy with the placement andready to start the simulation, she can say “Lock Patient.” Olivia says“Lock Patient,” the cursor, head, and pelvis guides disappear. Thesemi-transparent virtual patient guide overlay becomes opaque. Theindicator text changes to “Ready to Simulate.”

Olivia can glance down at the screen of the C&C app and can see anindicator that lets her know the patient has been registered and thatthe simulation is ready to run. She can remove the HoloLens and hand itto the first student, Stewie, to run through the simulation. The studentcan put on the HoloLens and confirm that he can see the virtual patientguide overlay running on the MRC app.

Back in the C&C app, Olivia can click the “Record Training” button. Thiscan create a video file from the HoloLens mixed reality view, as well asrecord the audio captured from the onboard microphones, each time theshe changes the patient condition and stage. The button turns green andstarts blinking, indicating that the mixed reality view from the MRC isbeing captured. Next, she selects “Tension Pneumothorax—Stab Wound” froma menu of available patient conditions. The app screen's UI updates with4 buttons, showing that this condition has 4 possible stages. She clicksbutton “Stage 1,” then clicks the “Start Simulation” button.

Inside the MRC app's viewport, the virtual patient guide overlay and“Ready to Simulate” text can disappear, being replaced with a realisticlooking virtual patient. Stewie walks around the mannequin and sees thatthe virtual patient's view changes as he moves around the mannequin. Theinstructor asks the student to describe what he's seeing and thinking ashe examines the patient. He kneels down and examines the left side ofthe virtual patient's chest and sees a shallow stab wound between the5th and 6th rib intercostal area. He scans the rest of the body anddoesn't see any other injuries. He watches the symmetrical rise and fallof virtual patient's chest as it breathes and checks the patient's pulseand blood oxygen levels. Stewie moves around the patient, describingwhat he's seeing, hearing and thinking. The camera and microphone on theHoloLens can capture what he's seeing and saying, streaming it from theMRC app to the DCS, which can store it for later evaluation.

Olivia moves to the second stage of the patient's condition by clickingthe “Stage 2” button on the C&C app, which can load the next piece ofpatient content in Stewie's MRC viewport. The virtual patient updates,exhibiting new visual and auditory cues including asymmetric rise andfall of the chest, the onset of cyanosis in the extremities, thin skintissues, and mucous membranes, as well as declining oxygen levels in theblood. Stewie scans different areas of the virtual patient, the hands,the pulse/ox sensor attached to the index finger, the veins around theneck, the coloration of lips, and the motions of the chest as thevirtual patient struggles to breath. He hears the patient's laboredbreathing. As Stewie sees and hears these cues he describes out loudwhat he's experiencing and thinking to Olivia. The MRC app can captureall of this and stream it to the DCS system.

The training session continues as Olvia steps through different stagesof the scenario on the C&C app, sometimes going back and forth betweenstages to allow Stewie to compare earlier cues to later cues to see theprogression in the MRC viewport. Eventually, the instructor concludesthe virtual patient simulation, toggling the “Record Training” buttonoff.

Such an example may be useful for basic stages of the training, wherethe program is heavily balanced toward instructor based teaching cues.When transitioning to the more advanced stages of practice it may beadvantageous to provide a list of SME-recognized critical cues on theC&C app to the instructor. This can allow the student to progress andmake mistakes without the instructor having to stop the session, or missout on student's actions, while taking notes.

For example, after splinting a fracture, standard practice is to checkpulse, motor function and sensation. If the student correctlyimmobilizes the injury in a simulation, but fails to reassess the pulse,the instructor could check the splint/sensation/motor function boxes,but could leave the reassess pulse box unchecked. Criticalcues/interventions that were missed could be tied to academic standardsof the student and/or lessons learned from real-world AARs, journals andarticles. It is contemplated that the HoloLens video recording be tiedwith instructor inputs. The SME recognized critical cues could be tiedto currently used pass/fail checklists and individual studentperformance reports could be generated to document training.

Numerous variations are contemplated, such as showing and hiding themannequin to allow for medical interventions, representing medicalinterventions on the virtual patient asset, and a streamlined fiducialmarker approach to patient registration. The DCS records for eachstudent can be set up by entering the student's name into the C&C appbefore starting a training scenario and toggling “Record Training”.Having multiple MRC clients involved in a training scenario mayincorporate multiple full equipment solutions and multiple instructors.Embodiments of the system and method can support a one-to-manyrelationship where a single C&C app and DCS system can support multipleHoloLens MRC clients. Embodiments of the system and method can use morethan one solution in an environment. Any suitable device registrationapproach can be used, where for example the system ties into eachdevice's unique MAC addresses, or the like, to facilitate.

Example after Action Review (AAR)

The instructor can review training videos on the Data Capture System (DCS) app. The instructor can scroll through a list of items, whichrepresent individual video files and session data captured from the MRCmixed reality viewport and onboard microphone. Each item in the list canhave the following information: Name of student, Name of instructor,Name of school, Scenario name, Scenario stage number, duration of videoclip, and date and timestamp when video was captured. The instructor canmove the mouse cursor over one of the items in the list and click it,expanding a video window. Playback controls can appear below the videowindow—Play, Pause, Rewind, Fast Forward and a Jog/Shuttle slider, forexample. The timecode for the video can be displayed above the playbackcontrols and can update as the video playhead moves. The instructor canhave the option of playing the video full screen.

In one embodiment, to make programming the DCS simpler, it may be usefulto watermark the info into the mixed reality video as an overlay in theMRC at runtime. The system can put the timestamp directly into the videoversus above the playback controls in one embodiment. Embodiments of thesystem and method can time stamp the instructor inputs onto theSME-checklist, which could correlate student action, instructor feedbackand performance standards into a precise student AAR.

Referring to FIG. 24, any suitable components and elements can beassociated with the MR client 180. Certain aspects of an example systemare described herein. An example system can include components as thosedescribed herein which can work together to provide a seamless learningexperience.

Mixed Reality Client (MRC)—Used by the student, the MRC 180 can be aWindows 10 application that can be loaded and running on a MicrosoftHoloLens. The MRC can provide the student with a view of the virtualpatient with minimal user interface elements.

Command & Control (C&C)—Used by the instructor, the C&C system 182 canbe a Windows 10 application that can loaded and running on a MicrosoftSurface tablet, for example. The MRC can connect to the C&C systemduring a simulation session, allowing the instructor to control thecontent, views, and states of the virtual patient.

Data Capture System (DCS)—The DCS 184 can be a Windows 10 applicationloaded and running on a laptop computer, for example. The DCS can beused to capture and store data from the MRC during a simulation session.The system can provide an interface for the instructor to browse andwatch video captured from the student's point-of-view during training.

Dedicated 5 GHz Wi-Fi Network (LAN)—Hardware associated with the systemcan connect together through a dedicated high-bandwidth LAN network 186.

Networking—Network Client of Command & Control (C&C) App

The networking setup for the HoloLens client can be made relatively easyand seamless for the end-user. The MRC can automatically connect to theLAN and to the C&C system if it's on the network, and can maintain somelevel of monitoring of these connections to make sure it's maintained.The system can connect to the C&C app's network server on the LANthrough a static address port. Alternatively, the system can listen forbroadcasts from the C&C app's network server, automatically connect toC&C app's address, and port dynamically. The system can receivevariables from the C&C app (strings, ints, floats, bools, structs,lists), receive commands from the C&C app (ex: set active scenariostage), send files to the C&C app (ex: JSON Scenario List & Scenarios),and/or send callbacks to the C&C app (ex: on change patient stage).

Networking—Network Server for Data Capture System (DCS)

The DCS system can connect with the HoloLens in order to receive videostreaming data as well as simple variables, such as the student's name,current training scenario and scenario stage being streamed. Theconnection between the DCS and MRC can be done automatically, with nointervention from the user, if both devices are on the network and theirrespective apps are up and running, for example. The system can allowthe DCS system to connect to the MRC app's network server on the LANthrough a static address. The system can associate with a broadcastnetwork server address and port through the LAN network, which can allowthe DCS system to connect dynamically to the MRC app. The system canfunction to send variables to the DCS app (ex: strings, ints, floats,bools, structs, lists), send commands to the DCS app (ex: prepare tocapture video stream), stream mixed reality audio and video to the DCSapp, and receive callbacks from the DCS app (ex: on video streamcaptured).

Data—Localized Content & Variables

Content for the MRC can be loaded from external data 190, such asexternal JSON files and deserialized in the HoloLens client app. Amaster Scenario List file 192 can contain local file path references tothe individual Scenario definition files. A Scenario definition file cancontain references to the local storage property 240 paths of namedcontent elements, such as Unity Asset Bundles, as well as runtimeproperties for the Scenario, such as the descriptive name of thescenario. These files can be used both by the MRC app, as well as theCommand & Control (C&C) app. Additionally, the MRC app can storevariables sent from the C&C app, which can then be passed along to theDCS, such as the name of the student and the name of the instructor. Thesystem can function to load JSON files used to build Scenario List (ex:of JSON Scenario files), load JSON files used to build a Scenario (ex:Scenario_Gunshot.json), load/store Unity Player Preferences used tomaintain pervasive application variables, receive student's name fromC&C app (ex: “John Doe”), receive instructor's name from C&C app (ex:“Jane Doe”), and receive school's name from C&C app (ex: “StateUniversity”).

Virtual Patient—Scenarios & Content

A Scenario 194 can contain a list of patient artwork assets that cancorrespond to individual Scenario Stages, as well as descriptive textthat can describe the content of the scenario. For instance, a tensionpneumothorax scenario might have four stages, thus there can be fourpatient artwork assets, one associated with each stage of the scenario.The descriptive text for this scenario might be: “Tension Pneumothorax:Knife Stab Wound.” The system can function to load a Scenario List whichcontains index of scenarios [JSON], load active scenarios which containreferences to virtual patient assets and total number of patient statesfor a current scenario [JSON, Unity Asset Bundles), manage states andviews of active scenarios, and load and present active virtual patientassets from scenarios.

Virtual Patient—Callbacks

The HoloLens client can communicate its states 198 and status back tothe C&C app throughout the course of a training session to keep thesystems in synch. This includes notifying the C&C when specific events212 occur, such as when the scenario has finished loaded, when thepatient for a scenario has successfully updated in the viewport, or whenan error occurs. This can allow the C&C app to keep track of the activestates on the MRC throughout a simulation session. The system canfunction to MRC connect/disconnect to C&C, DCS connect/disconnect toMRC, load Scenario List (ex: On MRC scenarios loaded, six scenariosavailable), load Scenario (ex: On MRC scenario loaded: Scenario.GSW),change Scenario, change Scenario Stage (ex: 1, 2, 3 . . . n), start/stopscenario, show/hide patient, mute/unmute patient audio, error (ex: OnMRC Error: “Scenario Not Found”).

Virtual Patient—2-Point Registration to Physical Mannequin

The process for registering the virtual patient to a physical mannequincan be simple and intuitive. This process may only need to be done onceat the beginning of a training session, by the instructor, before thestudent uses the MRC app. The registration process can involveleveraging the HoloLens' spatial mapping capabilities to place a virtualcursor on the surface of the mannequin's head and pelvis to set anchorreference points which are then used to position, orient, and offset thevirtual patient relative to the physical mannequin. The system can use a2-point manual registration process—Position, align, offset and placethe virtual patient relative to the physical mannequin through tworeference points (i.e. pelvis & head). The system can provideinstructional feedback to guide the user through the process ofregistering the virtual patient from start to finish. The system canprovide visual feedback of pelvis and head reference points in the MRCviewport, for example. The system can provide visual feedback of virtualpatient guide after reference points are set in the MRC viewport. Thesystem can show/hide visual feedback during registration process (ifdesired).

Mixed Reality Data Capture

The MRC can capture the mixed reality view of the student and virtualpatient together during the training session. The MRC can leverage theHoloLens' built-in video/audio capture and streaming capabilities tostore the student's learning performance, including what the student isseeing and hearing in real-time, as well as what they are saying andthinking as they describe their perceptions and thought processesthroughout the training session. The C&C app can provide the student'sname and instructor's name to the MRC, which can then pass along thisdata with the video/audio stream to the DCS system. The system canfunction to start/stop streaming video and audio of the mixed realityview to DCS app, send HoloLens unique machine ID to DCS app tofacilitate record-keeping, send student's name to DCS app (ex: “JohnDoe”), send instructor's name to DCS app (ex: “Jane Doe”), send school'sname to DCS app (ex: “The Farm”), send Scenario name to DCS app (ex:“Arterial Bleeding: Leg Amputation”), and send Scenario Stage number toDCS app (ex: 1, 2, 3 . . . n).

Commands—Respond Commands from the C&C App

The MRC can be remotely controlled by the instructor. The instructor cancontrol the student's entire training experience from the C&C app. Thesystem can be used to set MRC control modes (Local, Remote), set MRCruntime modes (Init, Register Patient, Ready, Simulate, Error), setactive Scenario, set active Scenario Stage, show/hide virtual patientart, mute/unmute patient audio, and start/stop streaming mixed realityview.

Commands—Local User Commands

If the instructor allows, there may be some simple MRC commands 214available to the student during a training session. These commands maynot have a visible user interface during training, but can be accessedthrough voice commands, for example. Example functionality can includeshow/hide patient view, which may be useful if the student needs to workwith the physical mannequin at any point during the training session)and mute/unmute patient audio.

System Feedback

The MRC can provide feedback to the user during the course of asimulation session when important state changes take place and whenproblems or errors occur. Examples include the loading of a newscenario, the changing of a scenario state, as well as problems withnetwork connectivity. This feedback may be toggled off and on by theinstructor. Example feedback can include MRC indicationconnected/disconnected to network, MRC indication connected/disconnectedto C&C, MRC indication DCS is connected/disconnected to MRC, MRCindication streaming mixed reality view to DCS, MRC indication ScenarioList loaded, MRC indication Scenario loaded, MRC indication Scenariostage changes, and MRC indication error occurred.

User Inputs

There can be multiple modes available for the student to interact withthe MRC systems. The student may do very little direct interactionthrough the system, as many or most of the features and content may becontrolled remotely by the instructor using the C&C app. That said,certain modes may be available to facilitate the patient-to-mannequinregistration process 218, setting up the network 220, debugging andtroubleshooting 224, as well as operating the MRC in solo mode andnavigating the virtual patient content without the C&C app. Examplefeatures 226 can include gaze inputs with visual cursor and gestureswhich can utilize HoloLens' spatial mapping and gaze tracking featuresthrough an sdk, speech inputs with voice commands which can useHoloLens' speech recognition through an sdk, gaze and speech inputs withvisual cursor and voice commands, network inputs with commands,callbacks and variables which connect to the network, send messages andvalues, and respond to network based callbacks, and system inputs withevents (ex: respond MRC systems events).

It will be appreciated that any suitable number of views 216 of userinterfaces are contemplated, including 3D views, hybrid views, and otheruser interface views. Example views and interfaces are provided below,with reference to FIG. 25 that describes a Mixed Reality Client (MRC)system.

Patient Scenario View

The MRC can provide a simple user interface (UI) to navigate andactivate a patient view 228 from a list of available scenario views 230.The UI can utilize gaze-based and/or voice based commands. Visual hintsmay need to be present to denote the corresponding voice commands.Functionality can include browse Scenario list and select Scenario.

Patient Scenario Stage View

The MRC can provide a simple UI to select a scenario stage from a numberof available scenario stages. The UI can utilize gaze-based and/or voicebased commands. Visual hints may need to be present denote thecorresponding voice commands. Functionality can include select ScenarioStage Number.

Patient 2-Point Overlay Registration Views

The MRC can provide a structured experience to guide the user throughregistering the virtual patient to the physical mannequin in a patientregistration view 238. This process can use voice commands, a visualcursor tied to the HoloLens' spatial mapping capabilities, and somefloating UI elements, for example. The UI can utilize both gaze-basedand voice based commands. Visual hints may need to be present to denotethe corresponding voice commands. Functionality can include RegisterPatient, where the user initiates the registration process, Place PelvisGuide, where the user places cursor on mannequin's torso, close to thepelvic floor, and locks a virtual pelvis guide into place, Place HeadGuide, where the user places cursor on mannequin's head and locks avirtual head guide into place, Reset Guides which can remove head,pelvis and virtual patient guides, if visible and resets system toinitial state, Lock Patient, where this option may be available afterthe head and pelvis guides are locked into place, and the virtualpatient guide appears, Show/Hide Virtual Patient Guide, where once bothhead and pelvis guides are placed, a semi-transparent Virtual PatientGuide appears, showing the corresponding position, alignment and offsetthat the virtual patient will have during the simulation. The user canshow/hide the Virtual Patient Guide to facilitate adjusting the head andpelvis guides, and Simulation Ready, where the user is finished with theregistration process and is ready to move onto simulating.

Networking View

The MRC can connect to the network and C&C app automatically as well asallow the DCS system to connect in as a background process. Duringtesting and development, and as a stop-gap in the event of networkproblems, this UI can be available to the user to manuallyconfigure/setup the network, connect to the LAN or C&C, or start the DCShosting services. The UI can utilize gaze-based and/or voice basedcommands. Visual hints may need to be present to denote thecorresponding voice commands. Functionality can include Show/HideNetwork View, Toggle connect to C&C automatically, Networking status,Display local device IP address & port, Input/Edit IP address and portof C&C Device, where the UI may have an onscreen keyboard or explicitvoice command entry for numbers, punctuation marks, “backspace” and“finished”, Connect/Disconnect C&C device, Start/Stop network server(for DCS), and Connect/Disconnect DCS device.

Voice Commands “Cheat Sheets” View

The MRC app can support a large number and variety of voice commands. Tofacilitate issuing these commands, a visual UI can provide a cheat-sheet232 of voice commands and corresponding keywords. On UI screens thatsupport voice commands outside of the scope of the “Cheat Sheet” UI,voice commands can be worked into the relevant UI. For example, on the“Patient Scenario Stage UI” the buttons to change the scenario stagenumber can also have hints for the corresponding voice commands—the setstage 1 button will have some visual clue, such as quotes around thebutton name “Set Stage 1”, to denote that it is also the voice command.Functionality can include Show/Hide Voice Command Hints View, Showavailable voice commands and keywords, and Navigate voice commands menus(next, previous, etc.).

Settings View

The MRC can allow the user to control some of the local app settings234. Some of these settings may be related to performance profiling,such as providing a view of the current frame-rate. Other settings canaffect the user experience, such as toggling the alerts and systemfeedback from the MRC app, as well as resetting the app's setting to thedefault values. Visual hints may need to be present to denote thecorresponding voice commands. Functionality can include Show/HideSettings View, Show/Hide system feedback, Show/Hide current studentname, Show/Hide frames per second, and Reset local Unity PlayerPreferences data.

Debug View

The MRC can provide a simple method for developers to see text-baseddebug and error log information at run-time within the compiled appwithout the need to be hooked up to a development system running afull-fledged debugger in the debugging view 224. In some embodiments,this may not replace a full-fledged debugger, but may provide somesimple functionality to help diagnose problems and errors within acompiled and deployed app. Visual hints may be present to denote thecorresponding voice commands. Functionality can include Show/Hide DebugUI, Show/Hide trace log, Show/Hide error log, Scroll trace log, andScroll error log.

The Data Capture System (DCS) 184 can include any suitable elements,features or functionality.

Networking—Network Client of Mixed Reality Client (MRC) App

The DCS system can be simple and intuitive to set up and can connect tothe LAN network 114 and to the MRC client. The process can be automated,with the DCS polling the network and connecting to the LAN and MRC onceit's up and running. The system can connect to the MRC app's networkserver on the LAN through a static address:port. The system can listenfor broadcasts from the MRC app's network server, automatically connectto MRC app's address:port dynamically. The system can function toreceive video/audio stream from the MRC app, receive variables from theMRC app (strings, ints, floats, bools, structs, lists), receive commandsfrom the MRC app, and send callbacks to the MRC app

Data

The DCS can store data locally on the host device's hard drive. Therecan be a localized database to facilitate organizing all of the trainingdata into browsable records. This database can be of standard format anddesign, with the ability to migrate the records and data into acloud-based storage solution. The video files can be accessible throughthe Windows File Manager, allowing the user to copy video files off ofthe device. The DCS can encode and create a file from the video/audiostream being received from the MRC app. The video codec format can bestandard and well supported, such as H264/MPEG-4 AVC. The amount ofcompression applied to the video file can balance quality with filesize, although the system can maximize the storage space on a laptop.Additional data can be captured from the MRC app, including the uniquehardware ID from the device sending the data, the name of the student,name of the instructor, name of the school, name of the Scenario andnumber of the scenario stage will need to be captured and stored aswell. The video can also be timestamped with the date/time it wascreated. All this data can be added to the metadata of the stored videofile, or at the very least, incorporated into the filename of the videofile. Functionality can include capture and store video/audio streamsfrom MRC, compute available storage space, capture and store scenariodata from MRC, capture name of student, name of instructor, name ofschool, scenario name, Scenario stage #, duration of video clip, dateand timestamp when video was captured, HoloLens device ID,date/Timestamp for when capture started, and date/timestamp for whencapture ended.

Commands

The functionality of the DCS can include browsing a list of studenttraining records, viewing the details of the records, and playing backvideo. Since there may be hundreds, if not thousands of records or more,there can be a simple mechanism to sort and filter the records to makeit easier for an instructor to hone in on the relevant data.Functionality can include Show/Hide active video stream from MRC,Show/Hide active student name using MRC, Create student record, Browselist of student records, Filter list of student records based onkeywords, Select student record in list, Control selected videoplayback, Play, Pause, Stop, Fast-forward, Rewind, Jog/Shuttle video,and Update current timecode.

System Feedback During Data Capture

The DCS can provide system feedback during a data capture session.Feedback can include notifications that the system has connected anddisconnected from the network and host MRC device, that the system iscurrently receiving video/data from the MRC, that the system hassuccessfully created and stored a student record, and when the systemencounters an error. Functionality can include Indication DCSconnected/disconnected from the network, Indication DCSconnected/disconnect from the MRC, Indication DCS is receivingvideo/data stream, Indication DCS has successfully captured and storedthe data and video stream, Indication DCS that storage space is runninglow, Indication DCS error occurred.

Callbacks to MRC

The DCS system can transmit states and status events back to the MRCapp. These callbacks 236 can include a notification when the DCS hassuccessfully connected to the MRC, DCS status at key times during avideo/data capture session, and when an error has occurred on the DCS.Functionality can include DCS connect to the MRC, DCS start receivingvideo stream, DCS finished receiving video stream, DCS video filesuccessfully created and stored from stream, and DCS error.

Inputs

The DCS system can have a few modes of input to control the system. Thefirst mode can be local, when the user is interacting with the system tobrowse and view records. This can be facilitated using a mouse orsimilar pointing device on the host system. The second mode can bethrough remote commands from the MRC system, which trigger actions andstream video/audio data and values through the network to the DCSsystem. Functionality can include touch inputs with user interactionsand network inputs with callbacks (i.e. change in MRC state).

It will be appreciated that the DCS can include any beneficial views oruser interfaces. Examples of such views and interfaces are describedherein.

Preview Capture Student Data View

During a video capture session, the DCS can provide the feedback thatvideo/data is successfully being captured. A preview of the video/datacan be displayed onscreen to verify the system is working as expected.Previewing functionality may affect performance, so there can be anoption to toggle it on and off, or pick a lower-resource intensivemethod. Functionality can include Show available storage space left,Show name of student, Show name of scenario, Show stage of scenario,Show live video, and Toggle video preview Off/On

Browse Student Records View

The DCS can provide a clean interface to browse and view studentrecords. Since there may be hundreds, if not thousands of records ormore, there may be a need for a mechanism to filter and sort theinformation to make it manageable for an instructor to quickly findspecific student records, as well as delete records and video files ifthey run out of storage space. Example functionality can include Showavailable storage space remaining, Scrollable list of student records,Scenario data visible in list view of student records, Student name,Date of record, Scenario name, Scenario stage, Input box to filter thelist view with keywords (ex—student name, scenario name, etc), Show/Hidevideo file associate with student record, and Delete record(s).

Watch Video File View

The DCS can provide a mechanism for the user to playback video files ina window and in fullscreen mode. There are control mechanisms that canallow the user to play, pause, stop, rewind, fast-forward the video, aswell as jog and shuttle the video using a slider. Functionality caninclude Play, Pause, Stop, Rewind, Fast-forward, Jog/Shuttle videoforward and backward, Update timecode of selected video as it changes,and Toggle fullscreen.

Settings UI

The DCS Settings UI can allow for setting properties that control thefunctionality of the app. Functionality can include Set video streamingparameters, Set video storage location, and Set video storage remainingwarning threshold.

Debug View

The DCS Debug UI can allow the user to display information that can beused to troubleshoot the deployed app. Functionality can includeShow/Hide Debug View, Show/Hide trace log, Scroll trace log, Show/Hideerror log, and Scroll error log.

Example Technical Specifications/Requirements

Certain non-limiting specifications and requirements for the system,method and apparatus of the disclosure are described below.

Mixed Reality Client (MRC) HoloLens App

The MRC app can be developed for Microsoft's HoloLens platform. TheHoloLens app can be installable through the Windows Store and throughside-loading (for development & testing). The HoloLens app can bedeveloped using Unity 5.5.2 (or newer) and C#. The HoloLens can useframeworks that are compatible with UWP/WSA features and functions thatcan work within the Unity development environment. Unity 5.5.2 currentlysupports .NET 2.0/3.5 Framework Profile and C# version 5. Final patientart and audio assets can be incorporated using Unity's AssetBundlesystem. Any custom networking services and features can be wrapped in C#for use in the Unity development environment.

Command & Control (C&C) App

The C&C app can be developed for a Microsoft Surface Pro style tablet(or comparable brand). The C&C app can be developed in Microsoft VisualStudio (or comparable/compatible IDE), and be deployed/managed usingMicrosoft's Windows Store or if possible, as web app. The C&C app can beinstallable through the Windows Store and through side-loading (fordevelopment & testing). The C&C app can share a screen with an existingapp which is used to interface and control the physical simulationmannequin hardware. The C&C app and the associated app can runside-by-side in a split screen mode configuration.

Data Capture System (DCS) App

The DCS app can be developed for a Windows 10 Laptop that can supportvideo capture and a large volume of internal data storage. The DCS appcan be developed in Microsoft Visual Studio (or comparable/compatibleIDE), and be deployed/managed using Microsoft's Windows Store, or ifpossible, as web app that can access localized storage and a DB forcaptured data and video content. A local database can be created on alaptop to store the captured data. Embodiments of the system and methodcan migrate this data online (cloud) as desired. The video codec usedfor encoding can be standard and common, not requiring problems withsharing the video outside the DCS system. For example, .H264/MPEG-AVC.The video files can be accessible outside the DCS system (throughWindows File Manager, for example). Embodiments of the system and methodcan allow a preview of the mixed reality video feed during capture.Portability of video data and database data is contemplated.

It will be appreciated that versions described herein can be scaled ormodified in any suitable manner. The solution can support multiple Wi-FiLAN, multiple HoloLens (MRC), multiple Surface Tablets (C&C), and/ormultiple Windows 10 Laptops (DCS). Multiple solutions can be operatingin the same environment in certain embodiments.

Networking Considerations

The network can be run on a dedicated 5G wireless LAN. The LAN may notbe connected to the internet, and may only serve to connect the variouslocal devices/machines in the training solution together. The devicescan operate within the optimal range of the wireless LAN. Embodiments ofthe system and method can use UDP, TCP or a combination of both fornetwork communications. UDP can probably be used for video/audiostreaming of content involved in capturing the mixed reality view fromthe HoloLens. UDP or TCP can be used for sending variables, messages andcommands between the Mixed Reality Client app, the Command & Control appand the Data Capture System app. Systems and configurations arecontemplated that work across all devices, provide reliablecommunications, keep network latency low, and use the least amount ofresources on each device.

Mixed Reality Capture on HoloLens

During mixed reality capture, the HoloLens system may down-throttle to amax of 30 Hz (i.e. 30 fps) refresh rate. This can degrade performanceand cause the experience to suffer (inducing judder and requiringlower-quality visuals). Any suitable approach to overcome this iscontemplated such as capturing low res video, or only capturing audio(if relevant) to reduce the load on the device. HoloLens may have a 5minute limit to the duration of video capture, which can be consideredwhen architecting the data capture approach, although systems withdifferent limits are contemplated.

Windows 10—Universal Windows Platform/Windows Store Apps (for Business)

It is believed that all systems authored for the Proteus solution canwork as native Windows 10 apps, to ensure hassle free interoperability,installation/upgrades, management, and maintenance, using Microsoft'sEnterprise tools.

It will be appreciated that a variety of systems, applications, andfeatures are contemplated. Examples of such features and systems caninclude the following:

1) Benchmarked database of trainee responses using naturalisticdecision-making (NDM) framework to train rapid recognition skills.Embodiments of the systems described herein can include session data(time stamped system actions, gaze tracking, video and audio) capturedduring a training session. Sessions can be built around scenarios thathave been constructed based on the special needs of the trainingpopulation (e.g., military populations place high value on airwayobstruction, tension pneumothorax injuries, and exsanguination types ofscenarios). The session data may include structured prompts to elicitspecific responses from trainees. These prompts can be developed basedon the NDM framework around which the entire tool has been developed.The actual trainee responses to the prompts, along with the session datathat is automatically captured (e.g., audio, video, gaze tracking) canbe aggregated and normed to provide meaningful benchmarks.

2) Biometric-based assessments of individual trainee and group traineeperformance. There has been widely disseminated research that has beendone showing the degradation curves of cognitive and physicalperformance that comes with increased physiologic stress. In certainembodiments, students can be outfitted with biometric tracking equipmentduring training to allow for assessments of specific cognitiveperformance as physiologic stress is increased. There are at least twopotentially important uses of such data. First, the scenario performancedata (e.g., speed to diagnose, proper identification of key cues,correct intervention chosen, etc.) can be paired with the biometric datafor any given student to highlight areas of deficiency and opportunitiesfor specific additional instruction. Second, the combination ofperformance and biometric data can be normed across all students whereit can be used for personnel selection (to aid inidentification/selection for advancement), and at the team/unit/grouplevel to create a clearer picture of total readiness.

3) Multi-point registration to match virtual and physical mannequin. Avirtual mannequin can overlaid across the top of the physical mannequinusing 2-point registration as described herein. The user can points areticule or cursor at the physical mannequin's head to lock in a headregistration point. The user can then place a second point at the pelvisof the physical mannequin and place another registration point. Theregistration can then be locked in place and the virtual mannequin canbe made visible, thus obscuring the physical mannequin. This multipointregistration concept can be extended in a variety of ways. For example,the user can use the reticule/cursor method to manually placeregistration points on any articulating joint of the physical mannequinand through a collection of registration points, improve the alignmentof the virtual mannequin with the physical mannequin. Alternatively,sensors can be placed on (or in) the physical mannequin that could bedetected by the registration module to lock in the precise location ofhead, torso, and all extremities. In addition to sensors that broadcastdata about the mannequin, registration could also be accomplished byplacing passive, fiducial markers at each articulating joint of thephysical mannequin, which could then be “seen” via computer vision andstored in the system. Embodiments of the system and method canincorporate a skeleton tracking capability and/or 3D scanning of themannequin (building a JSON-formatted archive file or .HAR file with alibrary of poses) to marry the virtual patient registration overlay datawith the physical mannequin tracking data. It will also be appreciatedthat augmented ultrasound is contemplated. For example, RFID chips canbe integrated onto/in the physical mannequin in anatomical locationsrepresenting underlying organs such as the lungs, heart, liver etc.Augmented reality can allow for creation of symptoms associated withnon-visible internal injuries such as hepatic hematoma, pleuraleffusion, and accumulation of free air in the thoracic cavity. An RFIDchip reader can be altered to meet the physical specifications of anultrasound machine and can give clinicians the ability to assess theirpatients in an accurate manner. Changes in the patient's condition canbe represented through various stages on the augmented ultrasoundmonitor screen. A NDM/SME framework can drive the student's cuerecognition and learning objectives.

4) Data capture using field-based, small computing systems. It may bebeneficial to provide a low-cost/low-power computing platform, (i.e. theRaspberry Pi or Edison), and create a low-level custom video/datacapture software. Such a system may provide a turn-key solution forother uses (e.g., small form factor, remote data capture capability).Instead of a Windows laptop, as described herein, a small networkableLinux device could be used in the field that runs on battery power. Toget data from the device, a number of options exist, including copyingit through a WIFI network, or utilizing the devices onboard USB ports.Some evaluation of the data can be performed locally and can feed datawirelessly back to the Command and Control module to further tailor thetraining experience (e.g., the trainee has only reviewed 3 of the 10relevant visual benchmarks on the patient). Embodiments of this devicecan be cloud-connected and allow for immediate transfer and be ready forsubsequent remote access and further analysis. Example devices can alsobe hooked up to a monitor and keyboard and used like a normal computer,to browse and review records.

5) Additional sensory input integrated into the system (e.g., scent).Embodiments of the virtual system, when paired with a physicalmannequin, can allow students to gain expertise through visual cues(e.g., mottled skin), audio cues (e.g., raspy breathing) and tactilecues (e.g., finding a bleeding artery and putting a tourniquet on it).Beyond these three senses additional senses can be stimulated. Forexample, the system can incorporate small canisters in and around thephysical mannequin to release scents during training sessions. Thesescents may be related to the condition itself (e.g., a chemical smellconsistent with a chemical burn injury) or may be just environmental(e.g., blood and vomit smells), to increase the level of stress andrealism of the training. The scent canisters can be controlled via theuser interface for the virtual system and can be timed to release scentat times that would be appropriate for various conditions that mightarise during a given scenario.

6) Tailored asset development and incorporation into the trainingsystem. Developing a sense of connection with simulated patients isoften challenging. Getting students to see the patient as more than justa piece of plastic (or a collection of pixels) can be an important partof creating training that is absorbed and retained. Embodiments includea portable tool that can allow administrators to capture images ofindividuals with whom the trainee has a connection (e.g., a member ofhis/her unit) and put that persons “face” and/or other characteristicsonto the virtual patient. Such a system can be small enough to travel toan on-site location where “local patient data” could be captured andintegrated into the system. Examples can build on existingphotogrammetry techniques already used to build the core model of thepatient, where the portable tool can incorporate highly customizedimages into the full 3-D model to personalize the patient model. Suchsystems may increase student engagement in the training process and maylead to better training outcomes.

7) Mechanical simulation integration. Embodiments of the system andmethod can capture the inputs provided to the physical simulationmannequin (e.g., SO2, respiration rate, heart rate) and incorporatethose inputs directly into the virtual patient where they may be used toalter the virtual patient's appearance or responses and/or be displayedon overlaid interventions (e.g., a virtual pulse oximeter on thefinger). A few, selected virtual patient states may be providedmanually, when the instructor requests them. But such Embodiments of thesystem and method can allow that process to happen automatically andwith a much greater number of virtual patient states.

8) Hand gesture detection to assess appropriateness of physicalinterventions. Embodiments of the system and method can capture thephysical interventions that students make on the physical patient. Oneembodiment would include building a recognition library of medical handgestures and then capturing student data using radar technology such asGoogle's Project Soli. Alternatively, the system could include asensor-based feedback system that could be embedded into the mannequinitself. A library of sensor feedback based on location and pressure canbe used to establish whether or not the student was making anappropriate medical intervention.

9) Mass casualty opportunities. Embodiments of the AR system can allowfor an extensible platform to include multiple wounded patients andsupporting contextual props (e.g., explosions, collapsed buildings,etc.) simultaneously. The system can include computer vision detectionof multiple systems operating in a geo-fenced area of operation.Physical mannequins and other elements in the environment can beBluetooth low-energy enabled to allow for detection by trainee wearablesystem. When a stimulus was detected, for example, the trainee canreceive presentation of the augmented reality stimulus via head-worndisplay. All systems can be tracked by a command and control system totrack all trainee's and their interaction with all stimuli.

10) Mixed reality—human moulage overlay on live people. Through the useof fiducial markers (e.g., images, stickers), Embodiments of the systemand method can present patient conditions that can be presented on livehumans.

11) NDM affordance, scaffolding. These tools include an adaptivetutoring framework to present multiple, highly salient cues to lessexperienced students and fewer, harder-to-detect cues for moreexperienced trainees. In addition, design elements can be provided thatserve as scaffolds to ensure that the student is oriented to see thepatient in a way that is consistent with the expert POV.

Some of the figures can include a flow diagram. Although such figurescan include a particular logic flow, it can be appreciated that thelogic flow merely provides an exemplary implementation of the generalfunctionality. Further, the logic flow does not necessarily have to beexecuted in the order presented unless otherwise indicated. In addition,the logic flow can be implemented by a hardware element, a softwareelement executed by a computer, a firmware element embedded in hardware,or any combination thereof.

The foregoing description of embodiments and examples has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or limiting to the forms described. Numerous modificationsare possible in light of the above teachings. Some of thosemodifications have been discussed, and others will be understood bythose skilled in the art. The embodiments were chosen and described inorder to best illustrate principles of various embodiments as are suitedto particular uses contemplated. The scope is, of course, not limited tothe examples set forth herein, but can be employed in any number ofapplications and equivalent devices by those of ordinary skill in theart. Rather it is hereby intended the scope of the invention to bedefined by the claims appended hereto.

We claim:
 1. A method for training in medical condition identificationand intervention, the method comprising the step of: providing awearable mixed reality viewing device, the wearable mixed realityviewing device comprising a non-planar viewing surface and a firstprocessor, the first processor comprising a first memory and a firstwireless transceiver; providing a computing device, the computing devicebeing in a wireless client-server based relationship with the wearablemixed reality viewing device, the computing device comprising asubstantially flat display screen, a second wireless transceiver, and asecond processor, the second processor comprising a second memory;providing a physical object, the physical object comprising a thirdprocessor and a third wireless transceiver, the physical object being inwireless communication with the computing device, and wherein thephysical object comprises moveable internal components being responsiveto wireless communications from the computing device to present visuallyperceived external movement of the physical object; storing a scenarioin the memory of the first processor, the scenario comprising parametersfor visualization of a medical condition in a virtual patient, themedical condition comprising a plurality of stages, each stagerepresenting a time-based state of the medical condition; storing thescenario in the memory of the second processor; anchoring a virtualimage of at least a portion of the virtual patient on at least a portionof the physical object, the virtual image being viewable on both thewearable mixed reality viewing device and the computing device;selecting the scenario, the scenario being transmitted as parameters ofthe medical condition portrayed by the virtual image to the firstprocessor of the wearable mixed reality viewing device and to the thirdprocessor of the physical object; commanding the second processor toinitiate displaying the virtual image in a selected one of the pluralityof stages, whereupon the virtual image is viewed in a three dimensionalaugmented reality view on the non-planar viewing surface of the wearablemixed reality viewing device and in two dimensions on the substantiallyflat display screen of the computing device, the virtual imagepresenting one or more virtual critical cues corresponding to thetime-based state of the medical condition; controlling one or morestates and one or more properties of the physical object to presentmovement of the physical object corresponding to one or more physicalcritical cues; assessing identification of the one or more virtualcritical cues; and assessing identification of the one or more physicalcritical cues.
 2. The method of claim 1, wherein the wearable mixedreality viewing device is head mounted and comprises a holographicprocessing unit with head-tracking capability.
 3. The method of claim 1,wherein the physical object is a mannequin.
 4. The method of claim 1,wherein the physical object is a portion of a mannequin.
 5. The methodof claim 1, wherein the physical object is a human mannequin and themovement is chest movement appearing as breathing.
 6. The method ofclaim 1, wherein the one or more virtual critical cues comprise cuesselected from the group consisting of skin color, vein appearance,bruise appearance, skin swelling, eye movement, mouth movement, blood,saliva, and combinations thereof.
 7. The method of claim 1, wherein theone or more physical critical cues comprise cues selected from the groupconsisting of breathing rate, symmetric and asymmetric chest movement,blood flow rate, pulse rate, and combinations thereof.
 8. The method ofclaim 1, wherein assessing a student's identification of the one or morevirtual critical cues comprises monitoring the augmented reality view onthe substantially flat display screen of the computing device and bygaze tracking determining that one or more of the one or more virtualcritical cues was viewed sufficiently to indicate identification.
 9. Asystem for training a student in medical condition identification andintervention, the system comprising: a wearable mixed reality viewingdevice, the wearable mixed reality viewing device comprising anon-planar viewing surface, a first wireless transceiver, and a firstprocessor, the first processor comprising a first memory and; acomputing device, the computing device being in a wireless client-serverbased relationship with the wearable mixed reality viewing device, thecomputing device comprising a substantially flat display screen, asecond wireless transceiver, and a second processor, the secondprocessor comprising a second memory; a physical object, the physicalobject being in a first location and comprising a third processor and athird wireless transceiver, the physical object being in wirelesscommunication with the computing device, and wherein the physical objectcomprises moveable internal components being responsive to wirelesscommunications from the computing device to present visually perceptiblemovement of the physical object; one or more fiducial markers placed ina predetermined position on a surface of the physical object; whereinone of the wearable mixed reality viewing device and the computingdevice comprises non-transitory computer-readable medium having inmemory executable instructions for: storing a fixed location in space,the fixed location being in known relationship to the one or morefiducial markers; translating one or more points from the one or morefiducial markers in a non-depth camera point-of-view from the wearablemixed reality viewing device to one or more anchor points related to a3D spatial map generated by the wearable mixed reality viewing device;anchoring a virtual image of at least a portion of a virtual patient tothe fixed location, the virtual image being viewable on both thewearable mixed reality viewing device and the computing device; storinga scenario in the memory of the first processor, the scenario comprisingparameters for visualization of the medical condition in the virtualpatient and comprising a plurality of stages, each stage representing atime-based state of the medical condition; storing the scenario in thememory of the second processor; selecting the scenario, the selectedscenario being transmitted as parameters of the virtual image to thefirst processor of the wearable mixed reality viewing device and to thethird processor of the physical object; commanding the second processorto initiate displaying the virtual image in a selected one of theplurality of stages, whereupon the virtual image is viewed in a threedimensional augmented reality view on the non-planar viewing surface ofthe mixed reality viewing device and in two dimensions on thesubstantially flat display screen of the computing device, the virtualimage presenting one or more virtual critical cues corresponding to thetime-based state of the medical condition; commanding the physicalobject to present movement corresponding to one or more physicalcritical cues of the time-based state of the medical condition;assessing a student's identification of the one or more virtual criticalcues; and assessing the student's identification of the one or morephysical critical cues.
 10. The system of claim 9, further comprisingexecutable instructions for, upon moving the physical object to a secondlocation, anchoring the virtual image to the physical object in thesecond location.
 11. The system of claim 9, wherein the mixed realityviewing device is head mounted and comprises a holographic processingunit with head-tracking capability.
 12. The system of claim 9, whereinthe physical object is a mannequin.
 13. The system of claim 9, whereinthe physical object is a portion of a mannequin and the one or morefiducial markers comprise an image selected from the group consisting ofan image of a tattoo.
 14. The system of claim 9, wherein the one or morevirtual critical cues comprise cues selected from the group consistingof skin color, vein appearance, bruise appearance, skin swelling, eyemovement, mouth movement, blood, saliva, and combinations thereof. 15.The system of claim 9, wherein the one or more physical critical cuescomprise cues selected from the group consisting of chest movement,pulse rate, and combinations thereof.
 16. The system of claim 9, whereinassessing the student's identification of the one or more virtualcritical cues comprises monitoring the augmented reality view on thesubstantially flat display screen of the computing device, thereby bygaze tracking determining that one or more of the one or more virtualcritical cues was viewed sufficiently to indicate identification.
 17. Asystem for training a student in medical condition identification andintervention, the system comprising: a wearable mixed reality viewingdevice, the wearable mixed reality viewing device comprising anon-planar viewing surface and a first processor, the first processorcomprising a first memory and a first wireless transceiver, the firstmemory storing a scenario, the scenario comprising parameters forvisualization of a medical condition in a virtual patient, the medicalcondition comprising a plurality of stages, each stage representing atime-based state of the medical condition, and wherein the virtualpatient presents one or more virtual critical cues corresponding to oneor more of the plurality of stages; a computing device, the computingdevice being in a wireless client-server based relationship with thewearable mixed reality viewing device, the computing device comprising asubstantially flat display screen and a second processor, the secondprocessor comprising a second memory and a second wireless transceiver,the computing device recording an assessment of a student'sidentification of the one or more virtual critical cues; a physicalobject, the physical object comprising a third processor and a thirdwireless transceiver, the physical object being in wirelesscommunication with the computing device, and wherein the physical objectcomprises moveable internal components being responsive to wirelesscommunications from the computing device to present visually perceptiblemovement of the physical object; wherein one of the wearable mixedreality viewing device and the computing device comprises non-transitorycomputer-readable medium having in memory executable instructions for:anchoring a virtual image of at least a portion of the virtual patienton at least a portion of the physical object, the virtual image beingviewable on both the wearable mixed reality viewing device and thecomputing device; selecting the scenario, the selected scenario beingtransmitted as parameters of the virtual image to the first processor ofthe wearable mixed reality viewing device and to the third processor ofthe physical object; initiating displaying the virtual image in aselected one of the plurality of stages, whereupon the virtual image isviewed in a three dimensional augmented reality view on the non-planarviewing surface of the wearable mixed reality viewing device and in twodimensions on the substantially flat display screen of the computingdevice; and initiating the physical object to present movementcorresponding to one or more physical critical cues of the time-basedstate of the medical condition.
 18. The system of claim 17, wherein themixed reality viewing device is head mounted and comprises a holographicprocessing unit with head-tracking capability.
 19. The system of claim17, wherein the physical object comprises at least a portion of amannequin.
 20. The system of claim 17, wherein the one or more virtualcritical cues comprise cues selected from the group consisting of skincolor, vein appearance, bruise appearance, skin swelling, eye movement,mouth movement, blood, saliva, and combinations thereof.